Nov 01 06:09:02.365637 ip-10-0-130-54 systemd[1]: Starting Kubernetes Kubelet... Nov 01 06:09:02.762514 ip-10-0-130-54 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:02.762514 ip-10-0-130-54 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 06:09:02.762514 ip-10-0-130-54 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:02.762514 ip-10-0-130-54 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:02.767247 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767185 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.767247 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767247 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.767247 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767250 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767259 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767262 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767264 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767267 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767270 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767273 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767276 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767278 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767281 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767283 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767286 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767288 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767290 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767293 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767296 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767298 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767301 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767303 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.767348 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767306 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767310 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767313 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767316 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767319 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767322 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767325 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767327 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767330 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767332 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767334 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767337 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767339 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767342 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767345 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767348 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767351 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767353 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767356 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.767785 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767359 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767361 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767364 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767366 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767368 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767371 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767373 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767376 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767392 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767395 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767397 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767399 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767402 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767404 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767407 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767409 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767413 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767416 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767419 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.768211 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767422 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767424 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767426 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767429 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767432 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767434 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767437 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767439 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767442 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767445 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767447 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767450 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767453 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767456 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767458 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767461 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767464 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767466 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767469 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767471 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.768660 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767474 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767476 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767479 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767481 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767486 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767489 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767491 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767494 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767496 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767499 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767501 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767503 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767506 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767508 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767510 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767513 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767516 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767518 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767522 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.769238 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767525 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767528 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767530 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767533 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767535 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767538 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.767540 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768277 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768284 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768287 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768290 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768293 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768295 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768298 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768300 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768303 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768306 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768308 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768312 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768316 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.769694 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768320 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768322 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768325 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768327 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768330 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768332 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768334 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768337 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768340 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768342 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768345 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768347 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768351 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768354 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768356 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768359 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768361 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768363 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768366 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768368 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.770192 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768370 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768373 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768375 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768393 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768397 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768399 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768401 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768404 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768406 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768409 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768411 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768414 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768416 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768419 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768421 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768425 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768428 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768431 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768433 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768435 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.770659 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768437 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768440 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768442 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768445 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768448 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768451 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768454 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768456 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768459 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768461 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768464 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768466 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768469 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768471 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768473 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768476 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768478 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768480 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768482 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768485 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.771159 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768488 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768490 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768493 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768495 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768497 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768500 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768502 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768504 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768507 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768509 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768512 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768515 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768517 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768519 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768522 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768524 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768527 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768530 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768532 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.771634 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768534 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768537 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768539 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768541 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768544 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768546 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768549 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768551 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768553 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768556 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768558 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768560 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768563 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.768565 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769008 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769012 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769015 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769019 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769023 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.772304 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769026 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769029 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769031 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769034 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769036 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769039 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769041 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769044 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769047 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769049 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769056 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769059 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769062 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769064 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769067 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769069 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769072 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769074 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769076 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769079 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.772799 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769081 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769084 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769086 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769088 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769091 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769093 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769095 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769098 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769100 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769102 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769105 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769107 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769109 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769112 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769114 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769116 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769119 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769121 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769123 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769126 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.773257 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769129 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769131 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769134 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769138 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769140 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769142 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769145 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769147 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769149 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769152 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769154 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769157 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769159 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769161 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769163 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769166 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769169 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769171 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769173 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.773727 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769176 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769179 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769181 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769184 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769186 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769188 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769192 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769195 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769197 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769199 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769202 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769204 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769207 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769209 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769211 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769215 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769217 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769220 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769222 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769225 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.774161 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769227 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769229 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769232 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769234 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769236 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769239 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769241 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769244 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769247 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769249 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769251 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769254 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769256 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769259 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769261 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769263 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769274 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769278 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769284 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.774630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769287 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769290 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.769293 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770248 2575 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770257 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770264 2575 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770269 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770273 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770277 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770281 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770285 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770288 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770291 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770295 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770298 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770301 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770304 2575 flags.go:64] FLAG: --cgroup-root="" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770306 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770309 2575 flags.go:64] FLAG: --client-ca-file="" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770312 2575 flags.go:64] FLAG: --cloud-provider="external" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770315 2575 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770318 2575 flags.go:64] FLAG: --cluster-domain="" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770321 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 06:09:02.775077 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770324 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770327 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770330 2575 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770335 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770337 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770340 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770343 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770347 2575 flags.go:64] FLAG: --contention-profiling="false" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770350 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770353 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770356 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770358 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770362 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770365 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770368 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770370 2575 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770373 2575 flags.go:64] FLAG: --enable-server="true" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770376 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770396 2575 flags.go:64] FLAG: --event-burst="100" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770399 2575 flags.go:64] FLAG: --event-qps="50" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770402 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770405 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770408 2575 flags.go:64] FLAG: --eviction-hard="" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770412 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 06:09:02.775609 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770415 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770418 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770420 2575 flags.go:64] FLAG: --eviction-soft="" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770423 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770426 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770429 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770431 2575 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770434 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770436 2575 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770439 2575 flags.go:64] FLAG: --feature-gates="" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770443 2575 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770446 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770449 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770452 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770455 2575 flags.go:64] FLAG: --healthz-port="10248" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770457 2575 flags.go:64] FLAG: --help="false" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770460 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770463 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770466 2575 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770469 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770472 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770475 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770478 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 06:09:02.776200 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770481 2575 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770484 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770487 2575 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770489 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770492 2575 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770495 2575 flags.go:64] FLAG: --kube-reserved="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770498 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770501 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770503 2575 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770506 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770509 2575 flags.go:64] FLAG: --lock-file="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770511 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770514 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770517 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770522 2575 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770525 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770527 2575 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770530 2575 flags.go:64] FLAG: --logging-format="text" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770533 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770536 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770538 2575 flags.go:64] FLAG: --manifest-url="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770541 2575 flags.go:64] FLAG: --manifest-url-header="" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770545 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770548 2575 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770551 2575 flags.go:64] FLAG: --max-pods="110" Nov 01 06:09:02.776759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770554 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770556 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770559 2575 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770562 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770564 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770567 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770570 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770576 2575 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770579 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770582 2575 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770585 2575 flags.go:64] FLAG: --pod-cidr="" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770588 2575 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770590 2575 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770593 2575 flags.go:64] FLAG: --pods-per-core="0" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770596 2575 flags.go:64] FLAG: --port="10250" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770599 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770602 2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0cc1c1bc772332679" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770605 2575 flags.go:64] FLAG: --qos-reserved="" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770607 2575 flags.go:64] FLAG: --read-only-port="10255" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770610 2575 flags.go:64] FLAG: --register-node="true" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770613 2575 flags.go:64] FLAG: --register-with-taints="" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770616 2575 flags.go:64] FLAG: --registry-burst="10" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770619 2575 flags.go:64] FLAG: --registry-qps="5" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770621 2575 flags.go:64] FLAG: --reserved-cpus="" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770624 2575 flags.go:64] FLAG: --reserved-memory="" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770627 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 06:09:02.777356 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770630 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770633 2575 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770635 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770638 2575 flags.go:64] FLAG: --runonce="false" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770641 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770644 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770647 2575 flags.go:64] FLAG: --seccomp-default="false" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770649 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770652 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770655 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770658 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770661 2575 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770663 2575 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770666 2575 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770671 2575 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770673 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770676 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770679 2575 flags.go:64] FLAG: --system-cgroups="" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770681 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770687 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770689 2575 flags.go:64] FLAG: --tls-cert-file="" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770692 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770696 2575 flags.go:64] FLAG: --tls-min-version="" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770698 2575 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770701 2575 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 06:09:02.777972 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770704 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770706 2575 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770709 2575 flags.go:64] FLAG: --v="2" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770713 2575 flags.go:64] FLAG: --version="false" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770717 2575 flags.go:64] FLAG: --vmodule="" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770720 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.770723 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770844 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770847 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770849 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770852 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770854 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770859 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770861 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770864 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770866 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770869 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770872 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770874 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770877 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770879 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.778556 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770883 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770885 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770888 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770891 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770895 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770898 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770909 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770913 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770915 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770918 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770921 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770923 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770926 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770928 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770930 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770933 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770936 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770938 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770941 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.779041 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770943 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770945 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770948 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770950 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770954 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770956 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770959 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770961 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770964 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770966 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770969 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770971 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770973 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770977 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770979 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770982 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770984 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770988 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770991 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.779495 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770993 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770996 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.770998 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771001 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771004 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771006 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771009 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771011 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771013 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771016 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771018 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771020 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771023 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771026 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771028 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771030 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771032 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771036 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771039 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771041 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.779969 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771043 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771046 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771048 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771050 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771053 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771055 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771059 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771061 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771063 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771066 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771068 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771070 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771073 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771075 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771078 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771080 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771082 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771085 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771087 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.780477 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771090 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771092 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771094 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771097 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771099 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771101 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771104 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771106 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771109 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771111 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771115 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771117 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771120 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771122 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771124 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.771149 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:02.780937 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771181 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771185 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771188 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771192 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771194 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771197 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771199 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771202 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771204 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771207 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771210 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771212 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771215 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771217 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771219 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771222 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771225 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771227 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771229 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.781300 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771232 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771234 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771236 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771239 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771241 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771244 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771246 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771248 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771252 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771255 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771257 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771259 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771262 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771264 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771267 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771269 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771273 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771275 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771278 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.781741 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771282 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771285 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771289 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771292 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771294 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771297 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771299 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771302 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771304 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771306 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771309 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771311 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771314 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771316 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771319 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771321 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771323 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771325 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771328 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.782168 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771330 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771333 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771336 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771339 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771341 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771345 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771347 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771350 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771352 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771354 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771358 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771360 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771363 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771365 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771369 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771372 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771375 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771390 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771393 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.782602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771395 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771398 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771400 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771403 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771405 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771408 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771413 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771416 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771418 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771421 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771423 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771426 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771428 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771430 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771434 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771436 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771439 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771441 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771444 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.783030 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771446 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771449 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771451 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771456 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771458 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771461 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771464 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771466 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771469 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.771471 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.777894 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.777909 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.778628 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.778633 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778687 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778692 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778695 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778698 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778701 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.783494 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778704 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778706 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778709 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778711 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778714 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778716 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778719 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778721 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778724 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778726 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778729 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778731 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778738 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778741 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778744 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778747 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778749 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778752 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778754 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778757 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.783927 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778760 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778762 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778765 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778768 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778771 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778773 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778776 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778778 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778781 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778783 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778786 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778788 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778791 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778793 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778795 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778798 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778800 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778802 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778805 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778807 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.784371 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778810 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778813 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778816 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778818 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778820 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778823 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778825 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778828 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778830 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778833 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778835 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778839 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778843 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778845 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778848 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778851 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778854 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778856 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778859 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.784834 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778861 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778864 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778866 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778870 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778873 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778876 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778878 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778881 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778883 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778886 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778888 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778891 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778894 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778896 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778898 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778907 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778912 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778915 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778917 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.785285 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778919 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778922 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778924 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778926 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778929 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778931 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778934 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778936 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778939 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778941 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778944 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778946 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778949 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778951 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778953 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778956 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778958 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778961 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778963 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778966 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.785775 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778968 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778970 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.778973 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.778999 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779134 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779138 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779141 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779144 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779146 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779149 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779151 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779154 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779156 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779159 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779161 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779163 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.786271 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779166 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779168 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779171 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779173 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779176 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779178 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779181 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779184 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779186 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779189 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779192 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779196 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779199 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779201 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779204 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779207 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779209 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779212 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779214 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.786738 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779216 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779219 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779221 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779224 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779226 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779228 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779231 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779234 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779237 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779239 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779241 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779244 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779246 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779248 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779251 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779253 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779255 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779258 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779261 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779263 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.787217 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779265 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779268 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779270 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779273 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779275 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779277 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779280 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779282 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779285 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779287 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779289 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779292 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779294 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779297 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779299 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779301 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779306 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779308 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.787689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779311 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779313 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779316 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779318 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779321 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779323 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779325 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779328 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779330 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779332 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779335 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779339 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779343 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779346 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779349 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779351 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779353 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779356 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779358 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.788355 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779360 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779363 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779365 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779368 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779370 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779372 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779374 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779377 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779397 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779400 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779403 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779406 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779408 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779411 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779413 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779416 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.789057 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779419 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.779443 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779474 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779478 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779481 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779483 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779486 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779488 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779491 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779493 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779495 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779498 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779501 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779503 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779506 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:02.789437 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779508 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779511 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779513 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779515 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779517 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779520 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779523 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779527 2575 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779530 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779533 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779536 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779538 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779540 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779543 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779545 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779547 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779550 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779553 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779555 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:02.789848 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779557 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779560 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779562 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779565 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779567 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779570 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779572 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779575 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779577 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779579 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779582 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779584 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779586 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779589 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779591 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779593 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779596 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779598 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779600 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779602 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:02.790606 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779605 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779607 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779609 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779612 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779614 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779616 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779619 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779621 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779624 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779626 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779629 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779631 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779634 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779636 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779640 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779643 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779646 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779649 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779651 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779654 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:02.791130 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779656 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779659 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779661 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779664 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779666 2575 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779668 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779671 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779673 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779675 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779677 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779680 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779682 2575 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779685 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779687 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779689 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779692 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779694 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779696 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779699 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779701 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:02.791602 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779704 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779706 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779709 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779711 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779714 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779716 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779719 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779721 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779723 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779726 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779729 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779731 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779734 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:02.779736 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.779794 2575 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.785234 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.786144 2575 server.go:1014] "Starting client certificate rotation" Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.786259 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 06:09:02.792040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.786305 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 06:09:02.792477 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.789456 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:02.792477 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.790781 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:02.802000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.801985 2575 log.go:25] "Validated CRI v1 runtime API" Nov 01 06:09:02.807683 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.807665 2575 log.go:25] "Validated CRI v1 image API" Nov 01 06:09:02.808504 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.808482 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 06:09:02.814178 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.814159 2575 fs.go:135] Filesystem UUIDs: map[1fef74f8-81ea-4e0f-a0ed-c0c7602b6ae8:/dev/nvme0n1p3 3cc278be-9e03-490b-aeef-313c90ff8c30:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 01 06:09:02.814237 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.814178 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 06:09:02.819046 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.818939 2575 manager.go:217] Machine: {Timestamp:2026-11-01 06:09:02.818492981 +0000 UTC m=+0.344685025 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3309702 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b1a7544627809171f7155ce03414 SystemUUID:ec28b1a7-5446-2780-9171-f7155ce03414 BootID:eab2a777-557f-4ea7-8677-b597860de589 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:37:9c:11:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:37:9c:11:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:32:35:26:97:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 06:09:02.819046 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.819040 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 06:09:02.819159 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.819101 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 06:09:02.822826 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.822801 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 06:09:02.822953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.822828 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-54.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 06:09:02.822993 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.822956 2575 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 06:09:02.822993 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.822965 2575 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 06:09:02.823052 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.822996 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:02.823052 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.823014 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:02.824142 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.824132 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 06:09:02.828477 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.828466 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 06:09:02.828587 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.828578 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 06:09:02.829417 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.829400 2575 kubelet.go:493] "Attempting to sync node with API server" Nov 01 06:09:02.829417 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.829422 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 06:09:02.829476 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.829432 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 06:09:02.829476 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.829441 2575 kubelet.go:405] "Adding apiserver pod source" Nov 01 06:09:02.829476 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.829448 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 06:09:02.829746 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.829735 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:02.835431 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.835414 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 06:09:02.841974 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.841961 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:02.842817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.842805 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 06:09:02.842854 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.842835 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 06:09:02.843022 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843011 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 06:09:02.843050 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843028 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 06:09:02.843050 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843035 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 06:09:02.843050 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843040 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 06:09:02.843050 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843045 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 06:09:02.843141 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843055 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 06:09:02.843744 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843736 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 06:09:02.843773 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843746 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 06:09:02.843773 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843752 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 06:09:02.843773 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843757 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 06:09:02.843773 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843771 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 06:09:02.843956 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.843780 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 06:09:02.844576 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.844566 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 06:09:02.844611 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.844579 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 06:09:02.848544 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.848529 2575 server.go:1263] "Started kubelet" Nov 01 06:09:02.848612 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.848577 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 06:09:02.849093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.849038 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 06:09:02.849196 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.849116 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 06:09:02.849320 ip-10-0-130-54 systemd[1]: Started Kubernetes Kubelet. Nov 01 06:09:02.850999 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.850981 2575 server.go:317] "Adding debug handlers to kubelet server" Nov 01 06:09:02.851280 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.851262 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 06:09:02.855719 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.855701 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 06:09:02.857040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.857029 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 06:09:02.857639 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.857625 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 06:09:02.857790 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.857765 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:02.858697 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.858582 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 06:09:02.858697 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.858575 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 06:09:02.858808 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.858701 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 06:09:02.858808 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.858792 2575 reconstruct.go:97] "Volume reconstruction finished" Nov 01 06:09:02.858808 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.858803 2575 reconciler.go:29] "Reconciler: start to sync state" Nov 01 06:09:02.859030 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.859017 2575 factory.go:55] Registering systemd factory Nov 01 06:09:02.859079 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.859034 2575 factory.go:223] Registration of the systemd container factory successfully Nov 01 06:09:02.860463 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.860444 2575 factory.go:153] Registering CRI-O factory Nov 01 06:09:02.860551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.860466 2575 factory.go:223] Registration of the crio container factory successfully Nov 01 06:09:02.860551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.860534 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 06:09:02.860648 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.860552 2575 factory.go:103] Registering Raw factory Nov 01 06:09:02.860648 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.860567 2575 manager.go:1199] Started watching for new ooms in manager Nov 01 06:09:02.862774 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.862758 2575 manager.go:319] Starting recovery of all containers Nov 01 06:09:02.871037 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.871020 2575 manager.go:324] Recovery completed Nov 01 06:09:02.872320 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.872294 2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 01 06:09:02.874758 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.874746 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:02.876116 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876095 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:02.876174 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876128 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:02.876174 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876139 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:02.876623 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876606 2575 cpu_manager.go:226] "Starting" policy="none" Nov 01 06:09:02.876623 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876621 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 06:09:02.876729 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876640 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 06:09:02.876729 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.876622 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:02.877320 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.877306 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:02.877399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.877335 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:02.877399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.877350 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:02.878753 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.878741 2575 policy_none.go:50] "Start" Nov 01 06:09:02.878814 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.878756 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 06:09:02.878814 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.878765 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 06:09:02.880711 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.880701 2575 policy_none.go:44] "Start" Nov 01 06:09:02.917842 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.917705 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.925705 2575 manager.go:341] "Starting Device Plugin manager" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.925726 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.925734 2575 server.go:87] "Starting device plugin registration server" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.925906 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.925914 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.926148 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.926234 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.926242 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.926703 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.926754 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.935046 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.936408 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.936423 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.936440 2575 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 06:09:02.941719 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.936469 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 06:09:02.988629 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:02.988610 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-54.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 06:09:02.988629 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.988609 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-54.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 06:09:02.989828 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:02.988601 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-54.us-west-1.compute.internal.18e3d6e204660a1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-54.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-54.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:02.848338458 +0000 UTC m=+0.374530502,LastTimestamp:2026-11-01 06:09:02.848338458 +0000 UTC m=+0.374530502,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-54.us-west-1.compute.internal,}" Nov 01 06:09:03.027055 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.027011 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:03.027740 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.027727 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:03.027802 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.027754 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:03.027802 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.027770 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:03.027802 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.027794 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.038223 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.038190 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal"] Nov 01 06:09:03.038278 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.038251 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:03.038917 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.038896 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:03.038970 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.038929 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:03.038970 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.038941 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:03.041192 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041178 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:03.041337 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.041402 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041358 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:03.041860 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041842 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:03.041947 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041865 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:03.041947 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041885 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:03.041947 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041870 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:03.041947 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041896 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:03.041947 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.041926 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:03.044093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.044077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.044177 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.044109 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:03.044737 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.044722 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:03.044822 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.044749 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:03.044822 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.044762 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:03.059492 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.059470 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 06:09:03.059578 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.059491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b0505d995be0782193889599bca279d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"2b0505d995be0782193889599bca279d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.059578 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.059537 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wnzkx" Nov 01 06:09:03.068864 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.068846 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.075140 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.075124 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.093545 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.093524 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 06:09:03.093545 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.093529 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 01 06:09:03.093695 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.093524 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 06:09:03.093695 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.093578 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-54.us-west-1.compute.internal.18e3d6e2060de33d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-54.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-54.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-54.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:02.876115773 +0000 UTC m=+0.402307816,LastTimestamp:2026-11-01 06:09:02.876115773 +0000 UTC m=+0.402307816,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-54.us-west-1.compute.internal,}" Nov 01 06:09:03.093695 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.093666 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-54.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 06:09:03.146121 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.146106 2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-54.us-west-1.compute.internal\" not found" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.159569 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.159550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.159650 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.159579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b0505d995be0782193889599bca279d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"2b0505d995be0782193889599bca279d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.159650 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.159594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.159755 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.159703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b0505d995be0782193889599bca279d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"2b0505d995be0782193889599bca279d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.259691 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.259671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.259759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.259696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.259759 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.259733 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.259826 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.259757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.294694 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.294655 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-54.us-west-1.compute.internal\" not found" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.346859 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.346843 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:03.347539 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.347526 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:03.347594 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.347553 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:03.347594 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.347563 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:03.347594 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.347582 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.373618 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.373597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.379596 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.379582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.404428 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.404411 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:03.404489 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.404430 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-54.us-west-1.compute.internal\": node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:03.513157 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.513140 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:03.613895 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.613855 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:03.714917 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.714890 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:03.785981 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:03.785964 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 06:09:03.786330 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.786089 2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae23e5d867b4645df8e862367584a87b-ec09ebbe119c1d65.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-54.us-west-1.compute.internal.18e3d6e2060e2a21\": read tcp 10.0.130.54:33138->18.117.54.58:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-54.us-west-1.compute.internal.18e3d6e2060e2a21 default 4840 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-54.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-54.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-54.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 06:09:02 +0000 UTC,LastTimestamp:2026-11-01 06:09:03.038935918 +0000 UTC m=+0.565127965,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-54.us-west-1.compute.internal,}" Nov 01 06:09:03.815239 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.815221 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:03.916032 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:03.915996 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:04.003529 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.003514 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:04.004160 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.004146 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:04.004231 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.004175 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:04.004231 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.004188 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:04.016893 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:04.016874 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:04.065723 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.065696 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 06:04:04 +0000 UTC" deadline="2028-07-12 17:46:16.755083692 +0000 UTC" Nov 01 06:09:04.065723 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.065722 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14867h37m12.689364277s" Nov 01 06:09:04.117571 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:04.117551 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:04.194887 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.194841 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:04.258254 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.258241 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:04.316124 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.316100 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:04.317998 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.317973 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:04.377520 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.377490 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:04.459360 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:04.459334 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0505d995be0782193889599bca279d.slice/crio-337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a WatchSource:0}: Error finding container 337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a: Status 404 returned error can't find the container with id 337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a Nov 01 06:09:04.459892 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:04.459875 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c62962b404022a5a6674917fdd042e4.slice/crio-a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd WatchSource:0}: Error finding container a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd: Status 404 returned error can't find the container with id a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd Nov 01 06:09:04.464614 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.464499 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:09:04.482941 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.482924 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:04.483110 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.483095 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:04.663188 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.663174 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:04.830739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.830701 2575 apiserver.go:52] "Watching apiserver" Nov 01 06:09:04.857875 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.857855 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 06:09:04.932280 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.932261 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 06:09:04.932564 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.932548 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-q9jbc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-kx8mt","kube-system/global-pull-secret-syncer-rj4mc","openshift-multus/multus-additional-cni-plugins-mdnbd","openshift-network-operator/iptables-alerter-5pdw4","kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-qmhjz","openshift-multus/multus-x5w6w","openshift-ovn-kubernetes/ovnkube-node-fk9cx","kube-system/konnectivity-agent-l576t","openshift-dns/node-resolver-v5cjs","openshift-multus/network-metrics-daemon-j7vht","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p"] Nov 01 06:09:04.936262 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.936246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:04.938916 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.938885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:04.939342 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:04.939294 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:04.941593 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.941575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:04.941683 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:04.941627 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:04.943725 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.943708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:04.943811 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:04.943762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:04.945805 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.945789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:04.947919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.947902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:04.950061 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.950043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x5w6w" Nov 01 06:09:04.952310 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.952296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:04.954519 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.954505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:04.956723 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:04.956707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.120862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.120846 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.123146 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.123130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.125556 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.125524 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"5c62962b404022a5a6674917fdd042e4","Type":"ContainerStarted","Data":"a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd"} Nov 01 06:09:06.125631 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.125570 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"2b0505d995be0782193889599bca279d","Type":"ContainerStarted","Data":"337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a"} Nov 01 06:09:06.128095 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-netns\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128169 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-config\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128169 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-env-overrides\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128244 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbm9\" (UniqueName: \"kubernetes.io/projected/6481f3ae-016a-4640-ac15-03e11f8963c6-kube-api-access-krbm9\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.128244 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysconfig\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.128244 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-tuned\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-serviceca\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-script-lib\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdtv\" (UniqueName: \"kubernetes.io/projected/d7014b11-e3e2-448c-b1bc-08f298821e3b-kube-api-access-9fdtv\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fms\" (UniqueName: \"kubernetes.io/projected/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-kube-api-access-m9fms\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-systemd\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.128467 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-host\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55368581-d897-48e9-bd96-95d33eba4426-iptables-alerter-script\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-systemd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-hostroot\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-multus-certs\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2x6\" (UniqueName: \"kubernetes.io/projected/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-kube-api-access-hl2x6\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-kubelet\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-bin\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-kubelet-config\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.128777 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128756 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-system-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-cnibin\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-etc-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-conf-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-etc-kubernetes\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-system-cni-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-ovn\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.128988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-dbus\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55368581-d897-48e9-bd96-95d33eba4426-host-slash\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-k8s-cni-cncf-io\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-socket-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dtbp\" (UniqueName: \"kubernetes.io/projected/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-kube-api-access-5dtbp\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-node-log\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cni-binary-copy\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-registration-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-sys\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-tmp\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-kubelet\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-daemon-config\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-modprobe-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-var-lib-kubelet\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-slash\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-log-socket\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-netd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nwm\" (UniqueName: \"kubernetes.io/projected/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-kube-api-access-x8nwm\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-netns\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-var-lib-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-bin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.129899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-multus\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-device-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-etc-selinux\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-host\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e862b42b-a7df-440b-9170-89c0de89c451-konnectivity-ca\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.129972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovn-node-metrics-cert\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cnibin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-lib-modules\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e862b42b-a7df-440b-9170-89c0de89c451-agent-certs\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nmn\" (UniqueName: \"kubernetes.io/projected/02d7b7a1-c649-45d0-9c18-14859b30ba56-kube-api-access-z8nmn\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-kubernetes\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-conf\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.130599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-systemd-units\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.131119 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snstj\" (UniqueName: \"kubernetes.io/projected/55368581-d897-48e9-bd96-95d33eba4426-kube-api-access-snstj\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.131119 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-socket-dir-parent\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.131119 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.131119 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-sys-fs\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.131119 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.130525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-run\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.150813 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.150795 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 06:09:06.150950 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.150800 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 06:09:06.151061 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.150820 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.151061 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.150859 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 06:09:06.151061 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.151011 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.151220 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.151063 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnjvc\"" Nov 01 06:09:06.151779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.151766 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 06:09:06.159181 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.159162 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 06:09:06.159263 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.159207 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 06:09:06.159263 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.159244 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 06:09:06.159263 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.159250 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.159390 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.159287 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.159390 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.159372 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwtj8\"" Nov 01 06:09:06.160629 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.160610 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdp5j\"" Nov 01 06:09:06.160700 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.160650 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.160700 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.160658 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 06:09:06.160700 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.160647 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.200363 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200348 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 06:09:06.200554 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200537 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 06:09:06.200610 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200594 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.200659 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200595 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.200659 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200611 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.200659 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200610 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnlbn\"" Nov 01 06:09:06.200659 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200621 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78q7p\"" Nov 01 06:09:06.200659 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200612 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.200824 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.200679 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m98z\"" Nov 01 06:09:06.221063 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.221046 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 06:09:06.230873 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snstj\" (UniqueName: \"kubernetes.io/projected/55368581-d897-48e9-bd96-95d33eba4426-kube-api-access-snstj\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.230953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-socket-dir-parent\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.230953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.230953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-sys-fs\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.230953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-run\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.230953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-netns\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230970 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-netns\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230992 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-socket-dir-parent\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.230996 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-sys-fs\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231023 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-run\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-config\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231061 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-env-overrides\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbm9\" (UniqueName: \"kubernetes.io/projected/6481f3ae-016a-4640-ac15-03e11f8963c6-kube-api-access-krbm9\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysconfig\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-tuned\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-serviceca\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b76995-eb22-418d-9665-bb729b6e88c4-tmp-dir\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.231165 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231164 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysconfig\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-script-lib\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdtv\" (UniqueName: \"kubernetes.io/projected/d7014b11-e3e2-448c-b1bc-08f298821e3b-kube-api-access-9fdtv\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fms\" (UniqueName: \"kubernetes.io/projected/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-kube-api-access-m9fms\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-systemd\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-host\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mh9f\" (UniqueName: \"kubernetes.io/projected/b2b76995-eb22-418d-9665-bb729b6e88c4-kube-api-access-2mh9f\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55368581-d897-48e9-bd96-95d33eba4426-iptables-alerter-script\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231426 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-systemd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-hostroot\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-multus-certs\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2x6\" (UniqueName: \"kubernetes.io/projected/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-kube-api-access-hl2x6\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-kubelet\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231563 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.231709 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231584 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-config\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-bin\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231647 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-systemd\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-hostroot\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231670 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-host\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-multus-certs\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231575 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-env-overrides\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231790 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-bin\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231816 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-systemd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-kubelet-config\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231872 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-kubelet\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-system-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-cnibin\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231913 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-kubelet-config\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-etc-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.232312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231951 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-system-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-cnibin\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-conf-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.231994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-etc-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-etc-kubernetes\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-system-cni-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-ovn\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-etc-kubernetes\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232039 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-conf-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.232049 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-dbus\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55368581-d897-48e9-bd96-95d33eba4426-host-slash\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232086 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-ovn\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.232117 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret podName:1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a nodeName:}" failed. No retries permitted until 2026-11-01 06:09:06.732096877 +0000 UTC m=+3.096395404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret") pod "global-pull-secret-syncer-rj4mc" (UID: "1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232133 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-dbus\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55368581-d897-48e9-bd96-95d33eba4426-host-slash\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.233127 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-k8s-cni-cncf-io\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232119 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-system-cni-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-socket-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232180 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-k8s-cni-cncf-io\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtbp\" (UniqueName: \"kubernetes.io/projected/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-kube-api-access-5dtbp\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-node-log\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cni-binary-copy\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232296 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-node-log\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-registration-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-sys\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-registration-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-tmp\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-kubelet\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232425 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-sys\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-daemon-config\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-modprobe-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.233884 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-var-lib-kubelet\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-slash\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232512 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-script-lib\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232571 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-slash\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232599 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-modprobe-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232630 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-kubelet\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-log-socket\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232689 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-log-socket\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-var-lib-kubelet\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-netd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cni-binary-copy\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232794 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-netd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232818 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nwm\" (UniqueName: \"kubernetes.io/projected/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-kube-api-access-x8nwm\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-netns\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-var-lib-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234436 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232881 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-daemon-config\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-bin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232911 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-netns\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232936 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-var-lib-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-multus\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232962 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-multus\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.232966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-device-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-etc-selinux\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233120 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-bin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-host\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233149 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e862b42b-a7df-440b-9170-89c0de89c451-konnectivity-ca\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.234933 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovn-node-metrics-cert\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-socket-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cnibin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-lib-modules\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233387 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e862b42b-a7df-440b-9170-89c0de89c451-agent-certs\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b76995-eb22-418d-9665-bb729b6e88c4-hosts-file\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nmn\" (UniqueName: \"kubernetes.io/projected/02d7b7a1-c649-45d0-9c18-14859b30ba56-kube-api-access-z8nmn\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-kubernetes\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-conf\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-systemd-units\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233587 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-lib-modules\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233685 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-kubernetes\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235460 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55368581-d897-48e9-bd96-95d33eba4426-iptables-alerter-script\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233727 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-host\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-device-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-conf\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233812 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cnibin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-systemd-units\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.233872 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-etc-selinux\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.233915 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.233990 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:06.733972111 +0000 UTC m=+3.098270629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.235537 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-tuned\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.235595 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-tmp\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.235919 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.235708 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovn-node-metrics-cert\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.251101 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251053 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wss87\"" Nov 01 06:09:06.251163 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251101 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 06:09:06.251163 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251116 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnk2x\"" Nov 01 06:09:06.251163 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251123 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.251163 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251103 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 06:09:06.251163 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251148 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.251385 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251228 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 06:09:06.251385 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251245 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 06:09:06.251385 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251265 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 06:09:06.251385 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251312 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vsdrw\"" Nov 01 06:09:06.251519 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.251504 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-serviceca\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.254775 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.254761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e862b42b-a7df-440b-9170-89c0de89c451-konnectivity-ca\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.256138 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.256124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e862b42b-a7df-440b-9170-89c0de89c451-agent-certs\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.285991 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.285914 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:06.285991 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.285942 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:06.285991 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.285955 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9qj2h for pod openshift-network-diagnostics/network-check-target-kx8mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:06.285991 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.286011 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h podName:62a9bf0c-6122-4683-80bf-43370ac319a4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:06.785994545 +0000 UTC m=+3.150293063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9qj2h" (UniqueName: "kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h") pod "network-check-target-kx8mt" (UID: "62a9bf0c-6122-4683-80bf-43370ac319a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:06.287569 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.287546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbm9\" (UniqueName: \"kubernetes.io/projected/6481f3ae-016a-4640-ac15-03e11f8963c6-kube-api-access-krbm9\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.288102 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.288062 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdtv\" (UniqueName: \"kubernetes.io/projected/d7014b11-e3e2-448c-b1bc-08f298821e3b-kube-api-access-9fdtv\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:06.288176 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.288161 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snstj\" (UniqueName: \"kubernetes.io/projected/55368581-d897-48e9-bd96-95d33eba4426-kube-api-access-snstj\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.288552 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.288535 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2x6\" (UniqueName: \"kubernetes.io/projected/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-kube-api-access-hl2x6\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.288675 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.288654 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fms\" (UniqueName: \"kubernetes.io/projected/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-kube-api-access-m9fms\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.288738 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.288722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nwm\" (UniqueName: \"kubernetes.io/projected/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-kube-api-access-x8nwm\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.288875 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.288854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtbp\" (UniqueName: \"kubernetes.io/projected/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-kube-api-access-5dtbp\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.289775 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.289757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nmn\" (UniqueName: \"kubernetes.io/projected/02d7b7a1-c649-45d0-9c18-14859b30ba56-kube-api-access-z8nmn\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.304589 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.304570 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 06:09:06.304675 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.304633 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6jnbx" Nov 01 06:09:06.333791 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.333769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b76995-eb22-418d-9665-bb729b6e88c4-tmp-dir\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.333864 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.333793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mh9f\" (UniqueName: \"kubernetes.io/projected/b2b76995-eb22-418d-9665-bb729b6e88c4-kube-api-access-2mh9f\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.333864 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.333840 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b76995-eb22-418d-9665-bb729b6e88c4-hosts-file\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.333927 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.333887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b76995-eb22-418d-9665-bb729b6e88c4-hosts-file\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.334051 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.334036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b76995-eb22-418d-9665-bb729b6e88c4-tmp-dir\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.388546 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.388520 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mh9f\" (UniqueName: \"kubernetes.io/projected/b2b76995-eb22-418d-9665-bb729b6e88c4-kube-api-access-2mh9f\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.428950 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.428928 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:06.431091 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.431075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:06.436204 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.436184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x5w6w" Nov 01 06:09:06.436995 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.436976 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55368581_d897_48e9_bd96_95d33eba4426.slice/crio-910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea WatchSource:0}: Error finding container 910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea: Status 404 returned error can't find the container with id 910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea Nov 01 06:09:06.439693 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.439631 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caa2f5d_b542_4af5_8dc4_3b75ab99c08f.slice/crio-3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5 WatchSource:0}: Error finding container 3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5: Status 404 returned error can't find the container with id 3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5 Nov 01 06:09:06.444403 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.444386 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:06.445050 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.445026 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0846b5f6_3f41_45a7_b3c4_0402f250d3c8.slice/crio-d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174 WatchSource:0}: Error finding container d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174: Status 404 returned error can't find the container with id d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174 Nov 01 06:09:06.449093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.449078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:06.451304 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.451222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:06.452711 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.452694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:06.452974 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.452953 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d1ba6a_d76e_4f87_84ec_3ddebb04b0cc.slice/crio-a08a35e430b13ae3cbe8b77384a2dc2da2a71c028437c5aedfdfa445afcdeaaa WatchSource:0}: Error finding container a08a35e430b13ae3cbe8b77384a2dc2da2a71c028437c5aedfdfa445afcdeaaa: Status 404 returned error can't find the container with id a08a35e430b13ae3cbe8b77384a2dc2da2a71c028437c5aedfdfa445afcdeaaa Nov 01 06:09:06.457839 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.457823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:06.459312 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.459263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:06.459574 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.459556 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d7b7a1_c649_45d0_9c18_14859b30ba56.slice/crio-4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd WatchSource:0}: Error finding container 4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd: Status 404 returned error can't find the container with id 4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd Nov 01 06:09:06.462668 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.462622 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8ebdf3_bc06_4ad6_b328_a4b1f973e9d1.slice/crio-6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368 WatchSource:0}: Error finding container 6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368: Status 404 returned error can't find the container with id 6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368 Nov 01 06:09:06.463416 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.463396 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6481f3ae_016a_4640_ac15_03e11f8963c6.slice/crio-502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1 WatchSource:0}: Error finding container 502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1: Status 404 returned error can't find the container with id 502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1 Nov 01 06:09:06.470132 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.470115 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode862b42b_a7df_440b_9170_89c0de89c451.slice/crio-ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965 WatchSource:0}: Error finding container ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965: Status 404 returned error can't find the container with id ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965 Nov 01 06:09:06.470929 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:06.470905 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b76995_eb22_418d_9665_bb729b6e88c4.slice/crio-5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473 WatchSource:0}: Error finding container 5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473: Status 404 returned error can't find the container with id 5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473 Nov 01 06:09:06.736603 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.736232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:06.736603 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.736324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:06.736603 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.736427 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:06.736603 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.736475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret podName:1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a nodeName:}" failed. No retries permitted until 2026-11-01 06:09:07.736462358 +0000 UTC m=+4.100760861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret") pod "global-pull-secret-syncer-rj4mc" (UID: "1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:06.736603 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.736524 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:06.736603 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.736549 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:07.736540753 +0000 UTC m=+4.100839254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:06.837527 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:06.837497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:06.837669 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.837638 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:06.837669 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.837654 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:06.837669 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.837665 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9qj2h for pod openshift-network-diagnostics/network-check-target-kx8mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:06.837826 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:06.837720 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h podName:62a9bf0c-6122-4683-80bf-43370ac319a4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:07.837704521 +0000 UTC m=+4.202003027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9qj2h" (UniqueName: "kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h") pod "network-check-target-kx8mt" (UID: "62a9bf0c-6122-4683-80bf-43370ac319a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:07.106311 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.106218 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l576t" event={"ID":"e862b42b-a7df-440b-9170-89c0de89c451","Type":"ContainerStarted","Data":"ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965"} Nov 01 06:09:07.108045 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.108018 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerStarted","Data":"502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1"} Nov 01 06:09:07.109550 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.109519 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5"} Nov 01 06:09:07.110788 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.110761 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5cjs" event={"ID":"b2b76995-eb22-418d-9665-bb729b6e88c4","Type":"ContainerStarted","Data":"5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473"} Nov 01 06:09:07.112323 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.112298 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9jbc" event={"ID":"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1","Type":"ContainerStarted","Data":"6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368"} Nov 01 06:09:07.113482 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.113460 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" event={"ID":"02d7b7a1-c649-45d0-9c18-14859b30ba56","Type":"ContainerStarted","Data":"4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd"} Nov 01 06:09:07.114539 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.114513 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" event={"ID":"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc","Type":"ContainerStarted","Data":"a08a35e430b13ae3cbe8b77384a2dc2da2a71c028437c5aedfdfa445afcdeaaa"} Nov 01 06:09:07.115610 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.115589 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x5w6w" event={"ID":"0846b5f6-3f41-45a7-b3c4-0402f250d3c8","Type":"ContainerStarted","Data":"d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174"} Nov 01 06:09:07.117367 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.117347 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pdw4" event={"ID":"55368581-d897-48e9-bd96-95d33eba4426","Type":"ContainerStarted","Data":"910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea"} Nov 01 06:09:07.309994 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.309953 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:06 +0000 UTC" deadline="2028-08-03 12:04:42.429590114 +0000 UTC" Nov 01 06:09:07.309994 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.309990 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15389h55m35.119604072s" Nov 01 06:09:07.744546 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.744511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:07.744711 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.744574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:07.744765 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.744714 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:07.744849 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.744779 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:09.744761395 +0000 UTC m=+6.109059927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:07.744919 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.744862 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:07.744919 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.744894 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret podName:1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a nodeName:}" failed. No retries permitted until 2026-11-01 06:09:09.744883932 +0000 UTC m=+6.109182437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret") pod "global-pull-secret-syncer-rj4mc" (UID: "1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:07.845158 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:07.845001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:07.845262 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.845203 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:07.845262 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.845225 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:07.845262 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.845236 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9qj2h for pod openshift-network-diagnostics/network-check-target-kx8mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:07.845458 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:07.845304 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h podName:62a9bf0c-6122-4683-80bf-43370ac319a4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:09.845267771 +0000 UTC m=+6.209566288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9qj2h" (UniqueName: "kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h") pod "network-check-target-kx8mt" (UID: "62a9bf0c-6122-4683-80bf-43370ac319a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:08.101858 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:08.101793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:08.102011 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:08.101930 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:08.102081 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:08.102034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:08.102128 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:08.102104 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:08.102180 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:08.102168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:08.102338 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:08.102300 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:08.311013 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:08.310966 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:06 +0000 UTC" deadline="2028-04-02 03:18:22.055505965 +0000 UTC" Nov 01 06:09:08.311013 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:08.311007 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12429h9m13.744503559s" Nov 01 06:09:09.130117 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.129415 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"2b0505d995be0782193889599bca279d","Type":"ContainerStarted","Data":"b3be458c781b12fe40dc9a966040f4fb647d361bac9d1539f88bbb378178f297"} Nov 01 06:09:09.145073 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.145043 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c62962b404022a5a6674917fdd042e4" containerID="b94189fd9a901225b1348266350d53741871a369bcc4f700cb9ef390c68434f7" exitCode=0 Nov 01 06:09:09.145203 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.145088 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"5c62962b404022a5a6674917fdd042e4","Type":"ContainerDied","Data":"b94189fd9a901225b1348266350d53741871a369bcc4f700cb9ef390c68434f7"} Nov 01 06:09:09.354216 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.354132 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" podStartSLOduration=4.354114793 podStartE2EDuration="4.354114793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:09.241266346 +0000 UTC m=+5.605564872" watchObservedRunningTime="2026-11-01 06:09:09.354114793 +0000 UTC m=+5.718413316" Nov 01 06:09:09.761082 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.760940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:09.761082 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.761006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:09.761307 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.761139 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:09.761307 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.761198 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:13.761181166 +0000 UTC m=+10.125479680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:09.761307 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.761262 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:09.761461 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.761310 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret podName:1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a nodeName:}" failed. No retries permitted until 2026-11-01 06:09:13.76129991 +0000 UTC m=+10.125598416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret") pod "global-pull-secret-syncer-rj4mc" (UID: "1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:09.861874 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:09.861561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:09.861874 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.861756 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:09.861874 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.861780 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:09.861874 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.861792 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9qj2h for pod openshift-network-diagnostics/network-check-target-kx8mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:09.861874 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:09.861844 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h podName:62a9bf0c-6122-4683-80bf-43370ac319a4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:13.861827018 +0000 UTC m=+10.226125537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9qj2h" (UniqueName: "kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h") pod "network-check-target-kx8mt" (UID: "62a9bf0c-6122-4683-80bf-43370ac319a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:10.099330 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.098809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:10.099330 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:10.098934 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:10.099330 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.099015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:10.099330 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:10.099091 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:10.099330 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.099160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:10.099330 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:10.099241 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:10.150572 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.150541 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"5c62962b404022a5a6674917fdd042e4","Type":"ContainerStarted","Data":"e35f4ce991f42b378ae2853f3be2935ce1c3e797090f60f4611592578a45c993"} Nov 01 06:09:10.156190 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.155391 2575 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="7f3ea5569716886b433acb9f4a30d9aab3e0412cb939cb6a01bb0d941316c7e7" exitCode=0 Nov 01 06:09:10.156190 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.156136 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerDied","Data":"7f3ea5569716886b433acb9f4a30d9aab3e0412cb939cb6a01bb0d941316c7e7"} Nov 01 06:09:10.379175 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:10.379070 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" podStartSLOduration=5.379053744 podStartE2EDuration="5.379053744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:10.263404733 +0000 UTC m=+6.627703254" watchObservedRunningTime="2026-11-01 06:09:10.379053744 +0000 UTC m=+6.743352269" Nov 01 06:09:12.101600 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:12.101175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:12.101600 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:12.101321 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:12.101600 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:12.101383 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:12.101600 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:12.101445 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:12.101600 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:12.101497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:12.101600 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:12.101559 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:12.169448 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:12.168693 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l576t" event={"ID":"e862b42b-a7df-440b-9170-89c0de89c451","Type":"ContainerStarted","Data":"2c18352a40e08be76ac6640b7da28dd3447630e1dca6658647d7912c7d73a170"} Nov 01 06:09:12.174411 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:12.174384 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9jbc" event={"ID":"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1","Type":"ContainerStarted","Data":"4ab8c724b207094308bb9f24a9da50e22221b0017d8902cae4d2f9e538daf5ef"} Nov 01 06:09:12.390962 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:12.390106 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l576t" podStartSLOduration=3.890529342 podStartE2EDuration="8.390085884s" totalImagesPullingTime="4.499556542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:12.279341207 +0000 UTC m=+8.643639736" watchObservedRunningTime="2026-11-01 06:09:12.390085884 +0000 UTC m=+8.754384406" Nov 01 06:09:13.180563 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:13.180525 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" event={"ID":"02d7b7a1-c649-45d0-9c18-14859b30ba56","Type":"ContainerStarted","Data":"7355025e37b71c00f5a3e6c244e899b8b5ae91e02c41c0689d7410466aa83df8"} Nov 01 06:09:13.792486 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:13.792449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:13.792671 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:13.792512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:13.792671 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.792626 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:13.792775 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.792699 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:21.792671623 +0000 UTC m=+18.156970137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:13.792775 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.792761 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:13.792930 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.792802 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret podName:1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a nodeName:}" failed. No retries permitted until 2026-11-01 06:09:21.792783952 +0000 UTC m=+18.157082459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret") pod "global-pull-secret-syncer-rj4mc" (UID: "1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:13.893454 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:13.893413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:13.893659 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.893637 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:13.893733 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.893666 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:13.893733 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.893680 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9qj2h for pod openshift-network-diagnostics/network-check-target-kx8mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:13.893853 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:13.893735 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h podName:62a9bf0c-6122-4683-80bf-43370ac319a4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:21.893717338 +0000 UTC m=+18.258015842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9qj2h" (UniqueName: "kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h") pod "network-check-target-kx8mt" (UID: "62a9bf0c-6122-4683-80bf-43370ac319a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:14.098822 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.098794 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:14.098957 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:14.098935 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:14.099026 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.099012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:14.099106 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:14.099085 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:14.103240 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.103217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:14.106661 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:14.103371 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:14.187161 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.187113 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x5w6w" event={"ID":"0846b5f6-3f41-45a7-b3c4-0402f250d3c8","Type":"ContainerStarted","Data":"e2166274c9a394a0b39fb759332f7a72c0736933eecd00bfaa5f5c2d6c44e444"} Nov 01 06:09:14.302170 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.302103 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-x5w6w" podStartSLOduration=2.687536403 podStartE2EDuration="10.302085644s" totalImagesPullingTime="7.614549241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:14.30160784 +0000 UTC m=+10.665906367" watchObservedRunningTime="2026-11-01 06:09:14.302085644 +0000 UTC m=+10.666384207" Nov 01 06:09:14.302703 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.302661 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q9jbc" podStartSLOduration=5.196328888 podStartE2EDuration="10.302649329s" totalImagesPullingTime="5.106320441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:12.390898955 +0000 UTC m=+8.755197478" watchObservedRunningTime="2026-11-01 06:09:14.302649329 +0000 UTC m=+10.666947851" Nov 01 06:09:14.851548 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:14.851529 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 06:09:15.098732 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.098630 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T06:09:14.85154392Z","UUID":"5cd17586-1868-4c65-b6c9-33d26ad1c701","Handler":null,"Name":"","Endpoint":""} Nov 01 06:09:15.102317 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.100105 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 06:09:15.102317 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.100134 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 06:09:15.192206 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.192132 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" event={"ID":"02d7b7a1-c649-45d0-9c18-14859b30ba56","Type":"ContainerStarted","Data":"4b50842fe61ef2c749fbe0157bcc80250878ceb6f73df1a0f7b031153f38f9ce"} Nov 01 06:09:15.194309 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.194264 2575 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="da099f88c0bd63b0341c99fd40bef1a7f0c56641aeef481aa79350ccae494ea8" exitCode=0 Nov 01 06:09:15.194409 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.194348 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerDied","Data":"da099f88c0bd63b0341c99fd40bef1a7f0c56641aeef481aa79350ccae494ea8"} Nov 01 06:09:15.195948 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.195924 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5cjs" event={"ID":"b2b76995-eb22-418d-9665-bb729b6e88c4","Type":"ContainerStarted","Data":"386231da0830701d08e25e25c0722e80b838a0931adf201e61b9925cccd9b486"} Nov 01 06:09:15.419478 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:15.419426 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v5cjs" podStartSLOduration=3.5033864980000002 podStartE2EDuration="11.41941378s" totalImagesPullingTime="7.916027282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:15.418952173 +0000 UTC m=+11.783250695" watchObservedRunningTime="2026-11-01 06:09:15.41941378 +0000 UTC m=+11.783712303" Nov 01 06:09:16.101137 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:16.100899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:16.101302 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:16.101239 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:16.101605 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:16.101584 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:16.101691 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:16.101674 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:16.101766 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:16.101731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:16.101815 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:16.101800 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:17.166087 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.166025 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:17.166621 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.166598 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:17.202190 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.202162 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" event={"ID":"02d7b7a1-c649-45d0-9c18-14859b30ba56","Type":"ContainerStarted","Data":"28799cd8fc99875fa6df9a46ef02ee56049dfd19d03245702b75802535f090a1"} Nov 01 06:09:17.203808 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.203783 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" event={"ID":"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc","Type":"ContainerStarted","Data":"16f443914b3206b9b0f8c023108054ee3ffc0f3ec96eecc5917a95917ba56b04"} Nov 01 06:09:17.205457 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.205434 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pdw4" event={"ID":"55368581-d897-48e9-bd96-95d33eba4426","Type":"ContainerStarted","Data":"fe170265dfa19cd7e550c6300d672888bf59117d9730fd857d7e9090c384139a"} Nov 01 06:09:17.388455 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.388403 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5pdw4" podStartSLOduration=4.806480089 podStartE2EDuration="13.388389711s" totalImagesPullingTime="8.581909622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:17.388163817 +0000 UTC m=+13.752462338" watchObservedRunningTime="2026-11-01 06:09:17.388389711 +0000 UTC m=+13.752688232" Nov 01 06:09:17.616366 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.616313 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" podStartSLOduration=3.155236811 podStartE2EDuration="13.616296951s" totalImagesPullingTime="10.46106014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:17.616210351 +0000 UTC m=+13.980508883" watchObservedRunningTime="2026-11-01 06:09:17.616296951 +0000 UTC m=+13.980595473" Nov 01 06:09:17.616853 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:17.616786 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" podStartSLOduration=3.520980347 podStartE2EDuration="13.616775625s" totalImagesPullingTime="10.095795278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:17.50473508 +0000 UTC m=+13.869033605" watchObservedRunningTime="2026-11-01 06:09:17.616775625 +0000 UTC m=+13.981074149" Nov 01 06:09:18.099315 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:18.099285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:18.099490 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:18.099290 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:18.099490 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:18.099449 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:18.099840 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:18.099823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:18.099949 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:18.099931 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:18.100025 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:18.100009 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:18.209400 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:18.209371 2575 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="50726883e5fafcccabbe487c2eda70d4f6af1d89d4f28d22b086c505f5cb542d" exitCode=0 Nov 01 06:09:18.209913 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:18.209473 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerDied","Data":"50726883e5fafcccabbe487c2eda70d4f6af1d89d4f28d22b086c505f5cb542d"} Nov 01 06:09:20.098901 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:20.098714 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:20.099577 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:20.098715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:20.099577 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:20.099004 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:20.099577 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:20.098715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:20.099577 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:20.099071 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:20.099577 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:20.099138 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:20.217439 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:20.217410 2575 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="7035e066194e0cf91e13571bf4ca725a08f410d0e4040b11cd7d41d896121039" exitCode=0 Nov 01 06:09:20.217598 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:20.217449 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerDied","Data":"7035e066194e0cf91e13571bf4ca725a08f410d0e4040b11cd7d41d896121039"} Nov 01 06:09:21.222605 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.222575 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"8e0ce4daecb80bfd2bbad135820d57e5e15c702322fa372b47c504b0bdaa36ce"} Nov 01 06:09:21.223150 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.222708 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"e7e3c7e1d2070caca7092fef25440f433b63628fc932a9c7b7042456a3bb06b1"} Nov 01 06:09:21.223150 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.222762 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"cafc8456fbdfe7845ecfc6f22347e1785237f43d12d74418488549836a820465"} Nov 01 06:09:21.223150 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.222806 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"dd08b1a289fb22d05d2a7cdbcfe68ca78212597066dae1c6390d8bf2a8b560a0"} Nov 01 06:09:21.223150 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.222850 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"71bd43a0e0066dd946daed9b37fd92a5d6dc667d07b53a6815da06b1714d8600"} Nov 01 06:09:21.223150 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.222891 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"17450cd4c0d9c660920c79ba859665332bbc8ef9a4c2f75c9984871be9b647a1"} Nov 01 06:09:21.858676 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.858455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:21.858825 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.858724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:21.858825 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.858591 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:21.858825 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.858797 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:37.858778063 +0000 UTC m=+34.223076579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 06:09:21.858825 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.858820 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:21.858979 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.858849 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret podName:1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a nodeName:}" failed. No retries permitted until 2026-11-01 06:09:37.858839863 +0000 UTC m=+34.223138365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret") pod "global-pull-secret-syncer-rj4mc" (UID: "1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a") : object "kube-system"/"original-pull-secret" not registered Nov 01 06:09:21.959335 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:21.959300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:21.959490 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.959471 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 06:09:21.959490 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.959489 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 06:09:21.959599 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.959501 2575 projected.go:196] Error preparing data for projected volume kube-api-access-9qj2h for pod openshift-network-diagnostics/network-check-target-kx8mt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:21.959599 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:21.959556 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h podName:62a9bf0c-6122-4683-80bf-43370ac319a4 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:37.95953589 +0000 UTC m=+34.323834403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9qj2h" (UniqueName: "kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h") pod "network-check-target-kx8mt" (UID: "62a9bf0c-6122-4683-80bf-43370ac319a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 06:09:22.099473 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:22.099159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:22.099473 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:22.099175 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:22.099473 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:22.099327 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:22.099473 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:22.099358 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:22.099473 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:22.099416 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:22.099473 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:22.099456 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:22.226465 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:22.226437 2575 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="f3d79c6b4980da5f6dd243b4c93883f0d6c7aa550e968d4ba6c2408c784189f6" exitCode=0 Nov 01 06:09:22.226726 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:22.226513 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerDied","Data":"f3d79c6b4980da5f6dd243b4c93883f0d6c7aa550e968d4ba6c2408c784189f6"} Nov 01 06:09:23.231357 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:23.231333 2575 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="449b5dbcfc0e08a6e52dacbfd45665d0bb1498d213e3d6f035c2881ce276c82a" exitCode=0 Nov 01 06:09:23.231827 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:23.231417 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerDied","Data":"449b5dbcfc0e08a6e52dacbfd45665d0bb1498d213e3d6f035c2881ce276c82a"} Nov 01 06:09:23.235239 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:23.235219 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"c888fb77a220d798e9e1165077f7c08bea645e52d440e2bdf46810405062dcf6"} Nov 01 06:09:24.102099 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:24.100493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:24.102099 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:24.100638 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:24.102099 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:24.101032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:24.102099 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:24.101137 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:24.102099 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:24.101197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:24.102099 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:24.101311 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:24.239747 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:24.239724 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" event={"ID":"6481f3ae-016a-4640-ac15-03e11f8963c6","Type":"ContainerStarted","Data":"f77a5616f68d09b13b23488281a685436bdb157efe8dfb72d3629e06b3edb84a"} Nov 01 06:09:24.451619 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:24.451562 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mdnbd" podStartSLOduration=7.527640741 podStartE2EDuration="20.451553634s" totalImagesPullingTime="12.923912893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:24.451470745 +0000 UTC m=+20.815769266" watchObservedRunningTime="2026-11-01 06:09:24.451553634 +0000 UTC m=+20.815852153" Nov 01 06:09:26.098764 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.098601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:26.099409 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.098601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:26.099409 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:26.098825 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:26.099409 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.098601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:26.099409 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:26.098907 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:26.099409 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:26.098988 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:26.247704 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.247677 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" event={"ID":"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f","Type":"ContainerStarted","Data":"bea0c8e2734302b1eb7f9d91dc8cddc81787b42d3902d5e5dc182270fd6557d3"} Nov 01 06:09:26.247977 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.247952 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:26.248085 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.248015 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:26.263119 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.263102 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:26.516742 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:26.516635 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" podStartSLOduration=8.157638078 podStartE2EDuration="22.516622705s" totalImagesPullingTime="14.358984627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:26.472469819 +0000 UTC m=+22.836768341" watchObservedRunningTime="2026-11-01 06:09:26.516622705 +0000 UTC m=+22.880921226" Nov 01 06:09:27.210140 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.210113 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:27.210582 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.210238 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 06:09:27.210637 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.210625 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:27.250487 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.250463 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:27.264643 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.264622 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:27.841122 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.841083 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rj4mc"] Nov 01 06:09:27.841342 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.841217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:27.841412 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:27.841344 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:27.843644 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.843620 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kx8mt"] Nov 01 06:09:27.843801 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.843718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:27.843866 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:27.843805 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:27.844253 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.844233 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j7vht"] Nov 01 06:09:27.844340 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:27.844330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:27.844438 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:27.844421 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:29.099417 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:29.099394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:29.099754 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:29.099425 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:29.099754 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:29.099471 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:29.099754 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:29.099521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:29.099754 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:29.099587 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:29.099754 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:29.099661 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:31.099172 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:31.099150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:31.099472 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:31.099159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:31.099472 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:31.099234 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rj4mc" podUID="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" Nov 01 06:09:31.099472 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:31.099159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:31.099472 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:31.099320 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kx8mt" podUID="62a9bf0c-6122-4683-80bf-43370ac319a4" Nov 01 06:09:31.099472 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:31.099407 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7vht" podUID="d7014b11-e3e2-448c-b1bc-08f298821e3b" Nov 01 06:09:32.894751 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:32.894620 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeReady" Nov 01 06:09:32.895042 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:32.894832 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 06:09:33.005854 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.005833 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lpzzd"] Nov 01 06:09:33.009240 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.009218 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cwffg"] Nov 01 06:09:33.009390 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.009371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:33.012364 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.012149 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-82lzc"] Nov 01 06:09:33.012504 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.012484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:33.013904 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.013883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.054871 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.054850 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tflw"] Nov 01 06:09:33.056158 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.056144 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-72qwz"] Nov 01 06:09:33.056309 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.056292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:33.057406 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.057390 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78b77c8f77-457bj"] Nov 01 06:09:33.057513 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.057498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.058808 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.058793 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-962j5"] Nov 01 06:09:33.058944 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.058921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.060076 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.060061 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p"] Nov 01 06:09:33.060175 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.060156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.061368 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.061350 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn"] Nov 01 06:09:33.061469 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.061456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:33.062911 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.062894 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fqwwj"] Nov 01 06:09:33.063000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.062986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.064229 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.064216 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:09:33.064359 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.064345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.065628 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.065610 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc"] Nov 01 06:09:33.065734 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.065722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.067979 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.067901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.098885 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.098867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:33.099213 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.099197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:33.099615 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.099596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:33.103899 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.103876 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.103992 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.103985 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 06:09:33.104047 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.104003 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.104998 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.104980 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl"] Nov 01 06:09:33.107112 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.107094 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d4d66c7f9-2g8gv"] Nov 01 06:09:33.107323 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.107308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.108770 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.108750 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd"] Nov 01 06:09:33.109393 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.109374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.113511 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.113489 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mkk52"] Nov 01 06:09:33.114569 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.114547 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:33.116993 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.116974 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2dxsg"] Nov 01 06:09:33.117105 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.117088 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.119009 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.118992 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lpzzd"] Nov 01 06:09:33.119101 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119025 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cwffg"] Nov 01 06:09:33.119101 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119045 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tflw"] Nov 01 06:09:33.119101 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119064 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-82lzc"] Nov 01 06:09:33.119101 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119083 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-72qwz"] Nov 01 06:09:33.119101 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119100 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p"] Nov 01 06:09:33.119385 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.119519 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.119500 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-962j5"] Nov 01 06:09:33.120335 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.120315 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78b77c8f77-457bj"] Nov 01 06:09:33.121025 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.121003 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mkk52"] Nov 01 06:09:33.121835 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.121817 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc"] Nov 01 06:09:33.122529 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.122511 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4d66c7f9-2g8gv"] Nov 01 06:09:33.123314 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.123298 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn"] Nov 01 06:09:33.145225 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gkc\" (UniqueName: \"kubernetes.io/projected/8079984d-980a-4287-826e-91194cf07e9e-kube-api-access-l5gkc\") pod \"kas-connection-checker-685895c7c9-cwffg\" (UID: \"8079984d-980a-4287-826e-91194cf07e9e\") " pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:33.145225 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145408 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5zp\" (UniqueName: \"kubernetes.io/projected/6e482476-c522-476f-ad22-ab83d135404c-kube-api-access-4v5zp\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.145408 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f7226f4-6e78-4172-8f28-c3225d93636f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.145408 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68vq\" (UniqueName: \"kubernetes.io/projected/cc06a215-884f-40b1-90fc-208bcc319a76-kube-api-access-l68vq\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.145551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthh4\" (UniqueName: \"kubernetes.io/projected/8fa19977-5c71-40c4-acb8-29e0d569efb4-kube-api-access-qthh4\") pod \"volume-data-source-validator-58c9f5f46d-4fn7p\" (UID: \"8fa19977-5c71-40c4-acb8-29e0d569efb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:33.145551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-serving-cert\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.145551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145551 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxck\" (UniqueName: \"kubernetes.io/projected/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-kube-api-access-wfxck\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cfef5efe-9658-4dbe-a3b5-cb484fec7962-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-snapshots\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8vr\" (UniqueName: \"kubernetes.io/projected/cfef5efe-9658-4dbe-a3b5-cb484fec7962-kube-api-access-8d8vr\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.145862 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rfc\" (UniqueName: \"kubernetes.io/projected/dd901c17-d465-4a03-b320-73be6ff3d0ce-kube-api-access-p9rfc\") pod \"kas-connection-checker-685895c7c9-6tflw\" (UID: \"dd901c17-d465-4a03-b320-73be6ff3d0ce\") " pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-config\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78c096fb-88d1-4f4c-813d-b2095302bf8c-config\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.145969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c8da36-c064-46e9-9712-b36bde260e3d-serving-cert\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78c096fb-88d1-4f4c-813d-b2095302bf8c-serving-cert\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-stats-auth\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-default-certificate\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47jb\" (UniqueName: \"kubernetes.io/projected/78c096fb-88d1-4f4c-813d-b2095302bf8c-kube-api-access-x47jb\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-service-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxns\" (UniqueName: \"kubernetes.io/projected/80c8da36-c064-46e9-9712-b36bde260e3d-kube-api-access-sdxns\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-tmp\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.148093 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.146647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrsr\" (UniqueName: \"kubernetes.io/projected/99cae588-1b8c-4972-bda1-781a3edb687a-kube-api-access-hnrsr\") pod \"kas-connection-checker-685895c7c9-lpzzd\" (UID: \"99cae588-1b8c-4972-bda1-781a3edb687a\") " pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:33.156782 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.156762 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fqwwj"] Nov 01 06:09:33.156880 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.156838 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd"] Nov 01 06:09:33.156880 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.156865 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl"] Nov 01 06:09:33.158463 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.158450 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:09:33.158542 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.158484 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2dxsg"] Nov 01 06:09:33.162606 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162572 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 06:09:33.162685 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162600 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.162991 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162805 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.162991 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162810 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.162991 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162891 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 06:09:33.162991 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162938 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.162991 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.162975 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163177 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163060 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163177 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163159 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163415 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163461 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163543 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163619 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163658 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163721 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163730 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163724 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163733 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163739 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163750 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163745 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163766 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163769 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163768 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.163787 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.163794 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 06:09:33.212860 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.212754 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 06:09:33.247216 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rfc\" (UniqueName: \"kubernetes.io/projected/dd901c17-d465-4a03-b320-73be6ff3d0ce-kube-api-access-p9rfc\") pod \"kas-connection-checker-685895c7c9-6tflw\" (UID: \"dd901c17-d465-4a03-b320-73be6ff3d0ce\") " pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:33.247318 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-config\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.247318 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.247318 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78c096fb-88d1-4f4c-813d-b2095302bf8c-config\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.247318 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.247318 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c8da36-c064-46e9-9712-b36bde260e3d-serving-cert\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.247559 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78c096fb-88d1-4f4c-813d-b2095302bf8c-serving-cert\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.247559 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.247559 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-stats-auth\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.247559 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-default-certificate\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.247559 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47jb\" (UniqueName: \"kubernetes.io/projected/78c096fb-88d1-4f4c-813d-b2095302bf8c-kube-api-access-x47jb\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-service-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxns\" (UniqueName: \"kubernetes.io/projected/80c8da36-c064-46e9-9712-b36bde260e3d-kube-api-access-sdxns\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-tmp\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.247779 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.247757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrsr\" (UniqueName: \"kubernetes.io/projected/99cae588-1b8c-4972-bda1-781a3edb687a-kube-api-access-hnrsr\") pod \"kas-connection-checker-685895c7c9-lpzzd\" (UID: \"99cae588-1b8c-4972-bda1-781a3edb687a\") " pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:33.248068 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.247904 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:33.248068 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.247962 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.747943914 +0000 UTC m=+30.112242418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:33.248068 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248050 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78c096fb-88d1-4f4c-813d-b2095302bf8c-config\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.248207 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248089 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-config\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.248207 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.248207 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gkc\" (UniqueName: \"kubernetes.io/projected/8079984d-980a-4287-826e-91194cf07e9e-kube-api-access-l5gkc\") pod \"kas-connection-checker-685895c7c9-cwffg\" (UID: \"8079984d-980a-4287-826e-91194cf07e9e\") " pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:33.248207 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5zp\" (UniqueName: \"kubernetes.io/projected/6e482476-c522-476f-ad22-ab83d135404c-kube-api-access-4v5zp\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f7226f4-6e78-4172-8f28-c3225d93636f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-tmp-dir\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248358 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-tmp\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248339 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-service-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.248399 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14525438-a3fb-4d7f-bd93-de2468852713-cert\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c66683c-48af-4ccd-9179-f61a555c3055-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgvz\" (UniqueName: \"kubernetes.io/projected/14525438-a3fb-4d7f-bd93-de2468852713-kube-api-access-kdgvz\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68vq\" (UniqueName: \"kubernetes.io/projected/cc06a215-884f-40b1-90fc-208bcc319a76-kube-api-access-l68vq\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthh4\" (UniqueName: \"kubernetes.io/projected/8fa19977-5c71-40c4-acb8-29e0d569efb4-kube-api-access-qthh4\") pod \"volume-data-source-validator-58c9f5f46d-4fn7p\" (UID: \"8fa19977-5c71-40c4-acb8-29e0d569efb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-serving-cert\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.248739 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42fs\" (UniqueName: \"kubernetes.io/projected/0e300938-7fc7-451b-9ca5-899929edea57-kube-api-access-b42fs\") pod \"network-check-source-6f78d59f99-bskfd\" (UID: \"0e300938-7fc7-451b-9ca5-899929edea57\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxck\" (UniqueName: \"kubernetes.io/projected/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-kube-api-access-wfxck\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.248989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f7226f4-6e78-4172-8f28-c3225d93636f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.249107 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cfef5efe-9658-4dbe-a3b5-cb484fec7962-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-snapshots\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8vr\" (UniqueName: \"kubernetes.io/projected/cfef5efe-9658-4dbe-a3b5-cb484fec7962-kube-api-access-8d8vr\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-config-volume\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssnx\" (UniqueName: \"kubernetes.io/projected/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-kube-api-access-kssnx\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249441 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c66683c-48af-4ccd-9179-f61a555c3055-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.249508 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.249465 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.74944674 +0000 UTC m=+30.113745263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:33.249912 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.249912 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.249912 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9xp\" (UniqueName: \"kubernetes.io/projected/6c66683c-48af-4ccd-9179-f61a555c3055-kube-api-access-9g9xp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.249912 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249732 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cfef5efe-9658-4dbe-a3b5-cb484fec7962-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.249912 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249752 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.249912 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.249787 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-snapshots\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.252951 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.252935 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 06:09:33.253028 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.252936 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 06:09:33.253028 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253006 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 06:09:33.253128 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253042 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 06:09:33.253218 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253143 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-q82mm\"" Nov 01 06:09:33.253265 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253232 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 06:09:33.253335 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253316 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 06:09:33.253401 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253382 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rmsht\"" Nov 01 06:09:33.253401 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253393 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rlspf\"" Nov 01 06:09:33.253496 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253382 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 06:09:33.253496 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253404 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 06:09:33.253496 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253446 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgqp9\"" Nov 01 06:09:33.253496 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253475 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ftfjm\"" Nov 01 06:09:33.253649 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253506 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 06:09:33.253649 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253551 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 06:09:33.253649 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253559 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 06:09:33.253649 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253581 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-v64qp\"" Nov 01 06:09:33.253649 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253619 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 06:09:33.253649 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253623 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tx6qb\"" Nov 01 06:09:33.253817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253720 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 06:09:33.253817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253743 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 06:09:33.253817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253748 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 06:09:33.253817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253756 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 06:09:33.253817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253760 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 06:09:33.253817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253802 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h2lht\"" Nov 01 06:09:33.254000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253834 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2658x\"" Nov 01 06:09:33.254000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253837 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 06:09:33.254000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253886 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jp6zc\"" Nov 01 06:09:33.254000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253921 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqrlq\"" Nov 01 06:09:33.254000 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.253973 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9n4lb\"" Nov 01 06:09:33.255642 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.255630 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7z9d7\"" Nov 01 06:09:33.255725 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.255711 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8r6tp\"" Nov 01 06:09:33.257854 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.257836 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:33.257953 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.257884 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.757873233 +0000 UTC m=+30.122171759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:33.258708 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.258691 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:33.258708 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.258709 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:33.258841 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.258746 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.758733893 +0000 UTC m=+30.123032418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:33.260045 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.259601 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:33.260045 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.259651 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.759638297 +0000 UTC m=+30.123936799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:33.260045 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.259725 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:33.260045 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.259775 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.759761532 +0000 UTC m=+30.124060034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:33.260045 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.259725 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.260547 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.260483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.260547 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.260497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.261795 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.261521 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bzrlb\"" Nov 01 06:09:33.261795 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.261548 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 06:09:33.261795 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.261625 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 06:09:33.261795 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.261653 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 06:09:33.261795 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.261693 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 06:09:33.261795 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.261761 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8pggw\"" Nov 01 06:09:33.262651 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.262198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78c096fb-88d1-4f4c-813d-b2095302bf8c-serving-cert\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.262651 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.262238 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-serving-cert\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.262651 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.262342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.262651 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.262609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-default-certificate\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.263444 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.263420 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c8da36-c064-46e9-9712-b36bde260e3d-serving-cert\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.263645 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.263627 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-stats-auth\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.263701 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.263660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.305035 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.304889 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rfc\" (UniqueName: \"kubernetes.io/projected/dd901c17-d465-4a03-b320-73be6ff3d0ce-kube-api-access-p9rfc\") pod \"kas-connection-checker-685895c7c9-6tflw\" (UID: \"dd901c17-d465-4a03-b320-73be6ff3d0ce\") " pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:33.306405 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.306385 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gkc\" (UniqueName: \"kubernetes.io/projected/8079984d-980a-4287-826e-91194cf07e9e-kube-api-access-l5gkc\") pod \"kas-connection-checker-685895c7c9-cwffg\" (UID: \"8079984d-980a-4287-826e-91194cf07e9e\") " pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:33.306806 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.306781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47jb\" (UniqueName: \"kubernetes.io/projected/78c096fb-88d1-4f4c-813d-b2095302bf8c-kube-api-access-x47jb\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.306880 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.306804 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthh4\" (UniqueName: \"kubernetes.io/projected/8fa19977-5c71-40c4-acb8-29e0d569efb4-kube-api-access-qthh4\") pod \"volume-data-source-validator-58c9f5f46d-4fn7p\" (UID: \"8fa19977-5c71-40c4-acb8-29e0d569efb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:33.307241 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.307218 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxck\" (UniqueName: \"kubernetes.io/projected/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-kube-api-access-wfxck\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.307463 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.307441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrsr\" (UniqueName: \"kubernetes.io/projected/99cae588-1b8c-4972-bda1-781a3edb687a-kube-api-access-hnrsr\") pod \"kas-connection-checker-685895c7c9-lpzzd\" (UID: \"99cae588-1b8c-4972-bda1-781a3edb687a\") " pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:33.307645 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.307626 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxns\" (UniqueName: \"kubernetes.io/projected/80c8da36-c064-46e9-9712-b36bde260e3d-kube-api-access-sdxns\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.307978 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.307959 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5zp\" (UniqueName: \"kubernetes.io/projected/6e482476-c522-476f-ad22-ab83d135404c-kube-api-access-4v5zp\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.308014 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.307973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68vq\" (UniqueName: \"kubernetes.io/projected/cc06a215-884f-40b1-90fc-208bcc319a76-kube-api-access-l68vq\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.308041 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.308009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.308635 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.308612 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.309386 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.309369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8vr\" (UniqueName: \"kubernetes.io/projected/cfef5efe-9658-4dbe-a3b5-cb484fec7962-kube-api-access-8d8vr\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.323170 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.323154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:33.329141 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.329124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:33.350704 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42fs\" (UniqueName: \"kubernetes.io/projected/0e300938-7fc7-451b-9ca5-899929edea57-kube-api-access-b42fs\") pod \"network-check-source-6f78d59f99-bskfd\" (UID: \"0e300938-7fc7-451b-9ca5-899929edea57\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:33.350810 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.350810 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.350810 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.350913 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.350913 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-config-volume\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.350913 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssnx\" (UniqueName: \"kubernetes.io/projected/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-kube-api-access-kssnx\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.350999 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c66683c-48af-4ccd-9179-f61a555c3055-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.350999 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.350999 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9xp\" (UniqueName: \"kubernetes.io/projected/6c66683c-48af-4ccd-9179-f61a555c3055-kube-api-access-9g9xp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.350999 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.350990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.351114 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.351140 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.351178 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-tmp-dir\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.351206 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14525438-a3fb-4d7f-bd93-de2468852713-cert\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.351237 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.351267 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c66683c-48af-4ccd-9179-f61a555c3055-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.351317 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgvz\" (UniqueName: \"kubernetes.io/projected/14525438-a3fb-4d7f-bd93-de2468852713-kube-api-access-kdgvz\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.351546 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.351516 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c66683c-48af-4ccd-9179-f61a555c3055-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.351914 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.350916 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:33.352002 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.351956 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.851938858 +0000 UTC m=+30.216237360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:33.352167 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.351568 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:33.352223 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.352172 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:33.352223 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.352190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-tmp-dir\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.352352 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.352230 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:33.852212106 +0000 UTC m=+30.216510633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:33.352465 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.352448 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.352688 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.352672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.352809 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.352780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-config-volume\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.353239 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.353219 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.354870 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.354849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.355124 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.355096 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14525438-a3fb-4d7f-bd93-de2468852713-cert\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.356098 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.356077 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c66683c-48af-4ccd-9179-f61a555c3055-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.358596 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.358576 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.367154 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.367132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:33.375383 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.375365 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:33.398742 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.398514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:33.407245 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.406037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:33.416855 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.416604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:33.418267 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.418240 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgvz\" (UniqueName: \"kubernetes.io/projected/14525438-a3fb-4d7f-bd93-de2468852713-kube-api-access-kdgvz\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.418911 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.418821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.418911 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.418892 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42fs\" (UniqueName: \"kubernetes.io/projected/0e300938-7fc7-451b-9ca5-899929edea57-kube-api-access-b42fs\") pod \"network-check-source-6f78d59f99-bskfd\" (UID: \"0e300938-7fc7-451b-9ca5-899929edea57\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:33.421492 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.421449 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.428805 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.428256 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssnx\" (UniqueName: \"kubernetes.io/projected/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-kube-api-access-kssnx\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.429951 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.429883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9xp\" (UniqueName: \"kubernetes.io/projected/6c66683c-48af-4ccd-9179-f61a555c3055-kube-api-access-9g9xp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.462040 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.461859 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:33.473783 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.473569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:33.486632 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.486419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:33.702523 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.702470 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lpzzd"] Nov 01 06:09:33.705725 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.705695 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cae588_1b8c_4972_bda1_781a3edb687a.slice/crio-656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa WatchSource:0}: Error finding container 656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa: Status 404 returned error can't find the container with id 656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa Nov 01 06:09:33.753934 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.753911 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tflw"] Nov 01 06:09:33.756620 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.756592 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd901c17_d465_4a03_b320_73be6ff3d0ce.slice/crio-99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c WatchSource:0}: Error finding container 99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c: Status 404 returned error can't find the container with id 99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c Nov 01 06:09:33.761223 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.761202 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-72qwz"] Nov 01 06:09:33.766667 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.766389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:33.766667 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.766433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.766667 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.766486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:33.766667 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.766544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:33.766667 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.766626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:33.766932 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.766756 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:33.766932 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.766848 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.766809825 +0000 UTC m=+31.131108349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:33.767146 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767098 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:33.767146 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.767131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767179 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.767163367 +0000 UTC m=+31.131461881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767301 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767345 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.767333273 +0000 UTC m=+31.131631788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767367 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767301 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767428 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.767418303 +0000 UTC m=+31.131716813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767381 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:33.768327 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.767495 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.767480995 +0000 UTC m=+31.131779508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:33.768948 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.768773 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.768751666 +0000 UTC m=+31.133050187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:33.793187 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.793045 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p"] Nov 01 06:09:33.797974 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.797939 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa19977_5c71_40c4_acb8_29e0d569efb4.slice/crio-719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a WatchSource:0}: Error finding container 719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a: Status 404 returned error can't find the container with id 719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a Nov 01 06:09:33.820043 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.820018 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cwffg"] Nov 01 06:09:33.823164 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.823139 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8079984d_980a_4287_826e_91194cf07e9e.slice/crio-b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b WatchSource:0}: Error finding container b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b: Status 404 returned error can't find the container with id b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b Nov 01 06:09:33.840071 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.840047 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-fqwwj"] Nov 01 06:09:33.843099 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.842849 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn"] Nov 01 06:09:33.843698 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.843558 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4526c0_30f7_4558_b9e6_4a949e19e8fb.slice/crio-18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc WatchSource:0}: Error finding container 18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc: Status 404 returned error can't find the container with id 18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc Nov 01 06:09:33.847038 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.847006 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c096fb_88d1_4f4c_813d_b2095302bf8c.slice/crio-cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696 WatchSource:0}: Error finding container cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696: Status 404 returned error can't find the container with id cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696 Nov 01 06:09:33.867848 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.867821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:33.867947 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.867881 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:33.868003 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.867973 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:33.868047 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.868031 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.868011382 +0000 UTC m=+31.232309899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:33.868097 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.868051 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:33.868097 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.868065 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:33.868181 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:33.868145 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:34.868128915 +0000 UTC m=+31.232427433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:33.887232 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.887210 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd"] Nov 01 06:09:33.892181 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.892160 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2dxsg"] Nov 01 06:09:33.892941 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:33.892917 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl"] Nov 01 06:09:33.899055 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.899037 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14525438_a3fb_4d7f_bd93_de2468852713.slice/crio-9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11 WatchSource:0}: Error finding container 9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11: Status 404 returned error can't find the container with id 9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11 Nov 01 06:09:33.899630 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:33.899402 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c66683c_48af_4ccd_9179_f61a555c3055.slice/crio-304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870 WatchSource:0}: Error finding container 304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870: Status 404 returned error can't find the container with id 304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870 Nov 01 06:09:34.266195 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.266163 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cwffg" event={"ID":"8079984d-980a-4287-826e-91194cf07e9e","Type":"ContainerStarted","Data":"eda28b13738b6f7846483d613b94d427764b3aac2053483945e91c29b96ccb3b"} Nov 01 06:09:34.266332 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.266198 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cwffg" event={"ID":"8079984d-980a-4287-826e-91194cf07e9e","Type":"ContainerStarted","Data":"b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b"} Nov 01 06:09:34.267380 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.267355 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2dxsg" event={"ID":"14525438-a3fb-4d7f-bd93-de2468852713","Type":"ContainerStarted","Data":"9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11"} Nov 01 06:09:34.268349 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.268333 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" event={"ID":"78c096fb-88d1-4f4c-813d-b2095302bf8c","Type":"ContainerStarted","Data":"cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696"} Nov 01 06:09:34.269420 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.269399 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-72qwz" event={"ID":"80c8da36-c064-46e9-9712-b36bde260e3d","Type":"ContainerStarted","Data":"822e37645d7dc97d8005d50d82b88d34a05a5c50d9703871d565a02c20deb59b"} Nov 01 06:09:34.270806 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.270787 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" event={"ID":"99cae588-1b8c-4972-bda1-781a3edb687a","Type":"ContainerStarted","Data":"c236f185d84017dfea469e2fe9ab1ad236dc42497f5cfe7bd540937d64645f63"} Nov 01 06:09:34.270892 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.270824 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" event={"ID":"99cae588-1b8c-4972-bda1-781a3edb687a","Type":"ContainerStarted","Data":"656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa"} Nov 01 06:09:34.271958 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.271940 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" event={"ID":"0e300938-7fc7-451b-9ca5-899929edea57","Type":"ContainerStarted","Data":"1050f3edd61abc487d1ca06a753631ea844d8800e5c6b1ef0590eca20622e9c1"} Nov 01 06:09:34.273053 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.273032 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" event={"ID":"6c66683c-48af-4ccd-9179-f61a555c3055","Type":"ContainerStarted","Data":"304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870"} Nov 01 06:09:34.274195 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.274179 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" event={"ID":"8fa19977-5c71-40c4-acb8-29e0d569efb4","Type":"ContainerStarted","Data":"719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a"} Nov 01 06:09:34.275174 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.275151 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" event={"ID":"1b4526c0-30f7-4558-b9e6-4a949e19e8fb","Type":"ContainerStarted","Data":"18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc"} Nov 01 06:09:34.276364 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.276346 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6tflw" event={"ID":"dd901c17-d465-4a03-b320-73be6ff3d0ce","Type":"ContainerStarted","Data":"3f494ddd2239ae8822f508e103ebc4a633dd4027372021b047b893203c8a5948"} Nov 01 06:09:34.276476 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.276374 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6tflw" event={"ID":"dd901c17-d465-4a03-b320-73be6ff3d0ce","Type":"ContainerStarted","Data":"99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c"} Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.776433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.776524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776549 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.776590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776631 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.776599223 +0000 UTC m=+33.140897728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776673 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776683 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776698 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776724 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.776708765 +0000 UTC m=+33.141007270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.776744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.776772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776832 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.776799861 +0000 UTC m=+33.141098369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.776868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776870 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:34.776902 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776874 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.776863419 +0000 UTC m=+33.141161950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:34.777733 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776912 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.776905078 +0000 UTC m=+33.141203579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:34.777733 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776921 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:34.777733 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.776964 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.776951774 +0000 UTC m=+33.141250294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:34.877793 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.877759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:34.877934 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:34.877827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:34.878087 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.878070 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:34.878150 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.878091 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:34.878150 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.878142 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.878126455 +0000 UTC m=+33.242424977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:34.878241 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.878224 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:34.878269 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:34.878257 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:36.878246502 +0000 UTC m=+33.242545005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:35.108203 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:35.108146 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cwffg" podStartSLOduration=419.108127723 podStartE2EDuration="6m59.108127723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:35.104113153 +0000 UTC m=+31.468411675" watchObservedRunningTime="2026-11-01 06:09:35.108127723 +0000 UTC m=+31.472426247" Nov 01 06:09:35.216203 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:35.216145 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6tflw" podStartSLOduration=419.216131943 podStartE2EDuration="6m59.216131943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:35.215679718 +0000 UTC m=+31.579978239" watchObservedRunningTime="2026-11-01 06:09:35.216131943 +0000 UTC m=+31.580430485" Nov 01 06:09:35.328852 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:35.328772 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" podStartSLOduration=419.328754202 podStartE2EDuration="6m59.328754202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:35.327269017 +0000 UTC m=+31.691567538" watchObservedRunningTime="2026-11-01 06:09:35.328754202 +0000 UTC m=+31.693052724" Nov 01 06:09:36.291949 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.291212 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" event={"ID":"8fa19977-5c71-40c4-acb8-29e0d569efb4","Type":"ContainerStarted","Data":"41671c9d988c6ce6e0354cb49a57552569e7fdcf98e7bc2dcc21ae7f2d1611c0"} Nov 01 06:09:36.403953 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.403866 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" podStartSLOduration=385.370594201 podStartE2EDuration="6m27.403848974s" totalImagesPullingTime="2.033254773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:36.402457379 +0000 UTC m=+32.766755902" watchObservedRunningTime="2026-11-01 06:09:36.403848974 +0000 UTC m=+32.768147495" Nov 01 06:09:36.797742 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.797696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:36.797923 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.797813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:36.797923 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.797843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:36.797923 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.797890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:36.798064 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.797943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:36.798114 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.798058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:36.798222 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798201 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:36.798222 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798225 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:36.798393 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798300 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.798265975 +0000 UTC m=+37.162564502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:36.798705 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798680 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:36.798784 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798737 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.798723495 +0000 UTC m=+37.163022009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:36.798838 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798820 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:36.798870 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798855 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.798844827 +0000 UTC m=+37.163143347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:36.798931 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798919 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.798910159 +0000 UTC m=+37.163208676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:36.798980 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798964 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:36.799039 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.798983 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.798977344 +0000 UTC m=+37.163275857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:36.799039 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.799014 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:36.799039 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.799031 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.799026327 +0000 UTC m=+37.163324839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:36.898839 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.898792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:36.899004 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:36.898857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:36.899004 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.898953 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:36.899111 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.899013 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.89899472 +0000 UTC m=+37.263293243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:36.899111 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.899035 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:36.899111 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.899057 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:36.899111 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:36.899110 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.899092492 +0000 UTC m=+37.263391011 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:37.911585 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:37.910502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:37.911585 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:37.910595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:37.911585 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:37.910814 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 06:09:37.911585 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:37.910870 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:10:09.910853481 +0000 UTC m=+66.275151994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : secret "metrics-daemon-secret" not found Nov 01 06:09:37.918031 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:37.917969 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:37.950825 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:37.950634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:38.012264 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.012228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:38.016211 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.016162 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:38.243472 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.243436 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:38.303903 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.303132 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-72qwz" event={"ID":"80c8da36-c064-46e9-9712-b36bde260e3d","Type":"ContainerStarted","Data":"3149b16f7c43e704038a18f7f5fcf42a196bc881d4c1f61b1bedee5c5590d6f8"} Nov 01 06:09:38.306248 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.306216 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/0.log" Nov 01 06:09:38.306392 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.306257 2575 generic.go:356] "Generic (PLEG): container finished" podID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" containerID="2cd0e117b3abb52829dcf0cf781dd4d1318cd5a3b7e7017c5aeb89e40dfa5379" exitCode=255 Nov 01 06:09:38.306392 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.306319 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" event={"ID":"1b4526c0-30f7-4558-b9e6-4a949e19e8fb","Type":"ContainerDied","Data":"2cd0e117b3abb52829dcf0cf781dd4d1318cd5a3b7e7017c5aeb89e40dfa5379"} Nov 01 06:09:38.306639 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.306618 2575 scope.go:122] "RemoveContainer" containerID="2cd0e117b3abb52829dcf0cf781dd4d1318cd5a3b7e7017c5aeb89e40dfa5379" Nov 01 06:09:38.361428 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.361405 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rj4mc"] Nov 01 06:09:38.372689 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:38.372664 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8e93ea_2cf0_4f69_ba21_ddf7c95fec9a.slice/crio-528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8 WatchSource:0}: Error finding container 528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8: Status 404 returned error can't find the container with id 528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8 Nov 01 06:09:38.418167 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.418108 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-72qwz" podStartSLOduration=385.474687333 podStartE2EDuration="6m29.418089809s" totalImagesPullingTime="3.943402476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:38.416598187 +0000 UTC m=+34.780896709" watchObservedRunningTime="2026-11-01 06:09:38.418089809 +0000 UTC m=+34.782388330" Nov 01 06:09:38.641730 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.641697 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kx8mt"] Nov 01 06:09:38.752407 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:38.752376 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a9bf0c_6122_4683_80bf_43370ac319a4.slice/crio-bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee WatchSource:0}: Error finding container bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee: Status 404 returned error can't find the container with id bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee Nov 01 06:09:38.902811 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.902782 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zf9qt"] Nov 01 06:09:38.921044 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.921018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:38.963541 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.963516 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zf9qt"] Nov 01 06:09:38.973554 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.973530 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 06:09:38.974589 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.973846 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wr\"" Nov 01 06:09:38.974751 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:38.973954 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 06:09:39.023674 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.023650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-crio-socket\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.023823 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.023808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.024159 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.024135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-data-volume\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.024614 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.024461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgwv\" (UniqueName: \"kubernetes.io/projected/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-api-access-bmgwv\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.024614 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.024566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125233 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-data-volume\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125384 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgwv\" (UniqueName: \"kubernetes.io/projected/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-api-access-bmgwv\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125384 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125503 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.125404 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:39.125503 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.125457 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:39.625440769 +0000 UTC m=+35.989739284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : configmap "kube-rbac-proxy" not found Nov 01 06:09:39.125599 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-crio-socket\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125647 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-data-volume\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125647 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125728 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.125613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-crio-socket\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.125728 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.125690 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:39.125783 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.125744 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:39.625729553 +0000 UTC m=+35.990028067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:39.181484 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.181459 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgwv\" (UniqueName: \"kubernetes.io/projected/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-api-access-bmgwv\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.312401 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.312355 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2dxsg" event={"ID":"14525438-a3fb-4d7f-bd93-de2468852713","Type":"ContainerStarted","Data":"751c307ca4a011137367641f06fe518c63abeb448cd8f0639fc91ceacf9bd041"} Nov 01 06:09:39.314593 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.314565 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" event={"ID":"78c096fb-88d1-4f4c-813d-b2095302bf8c","Type":"ContainerStarted","Data":"762900f1bce1250ca4d3c1757985a6571a52a22be2f2dfbf2ee11d7f96a1b075"} Nov 01 06:09:39.315817 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.315791 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kx8mt" event={"ID":"62a9bf0c-6122-4683-80bf-43370ac319a4","Type":"ContainerStarted","Data":"bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee"} Nov 01 06:09:39.317039 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.316995 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rj4mc" event={"ID":"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a","Type":"ContainerStarted","Data":"528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8"} Nov 01 06:09:39.322036 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.322011 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" event={"ID":"6c66683c-48af-4ccd-9179-f61a555c3055","Type":"ContainerStarted","Data":"9eea78d72801a7783d553e90c312769408b1d16644eba546c0af7762af56b6b9"} Nov 01 06:09:39.325117 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.325089 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/1.log" Nov 01 06:09:39.325734 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.325686 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/0.log" Nov 01 06:09:39.325734 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.325725 2575 generic.go:356] "Generic (PLEG): container finished" podID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" exitCode=255 Nov 01 06:09:39.325875 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.325856 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" event={"ID":"1b4526c0-30f7-4558-b9e6-4a949e19e8fb","Type":"ContainerDied","Data":"df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca"} Nov 01 06:09:39.325934 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.325902 2575 scope.go:122] "RemoveContainer" containerID="2cd0e117b3abb52829dcf0cf781dd4d1318cd5a3b7e7017c5aeb89e40dfa5379" Nov 01 06:09:39.326124 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.326109 2575 scope.go:122] "RemoveContainer" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" Nov 01 06:09:39.326297 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.326255 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fqwwj_openshift-console-operator(1b4526c0-30f7-4558-b9e6-4a949e19e8fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" Nov 01 06:09:39.424013 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.423926 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2dxsg" podStartSLOduration=2.155023409 podStartE2EDuration="7.423909949s" totalImagesPullingTime="5.26888654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:39.423171454 +0000 UTC m=+35.787469975" watchObservedRunningTime="2026-11-01 06:09:39.423909949 +0000 UTC m=+35.788208475" Nov 01 06:09:39.535596 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.535523 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" podStartSLOduration=385.55423115 podStartE2EDuration="6m30.535502445s" totalImagesPullingTime="4.981271295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:39.534477685 +0000 UTC m=+35.898776207" watchObservedRunningTime="2026-11-01 06:09:39.535502445 +0000 UTC m=+35.899800968" Nov 01 06:09:39.632053 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.632017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.632198 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.632081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:39.632243 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.632221 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:39.632297 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.632268 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.632255379 +0000 UTC m=+36.996553882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:39.632570 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.632556 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:39.632608 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:39.632598 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:40.632587521 +0000 UTC m=+36.996886023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : configmap "kube-rbac-proxy" not found Nov 01 06:09:39.759081 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:39.758990 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" podStartSLOduration=385.83987256 podStartE2EDuration="6m30.758974599s" totalImagesPullingTime="4.919102039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:39.757028922 +0000 UTC m=+36.121327446" watchObservedRunningTime="2026-11-01 06:09:39.758974599 +0000 UTC m=+36.123273120" Nov 01 06:09:40.334454 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.334270 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/1.log" Nov 01 06:09:40.335809 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.335787 2575 scope.go:122] "RemoveContainer" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" Nov 01 06:09:40.335967 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.335945 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fqwwj_openshift-console-operator(1b4526c0-30f7-4558-b9e6-4a949e19e8fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" Nov 01 06:09:40.558658 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.557548 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g"] Nov 01 06:09:40.571072 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.570805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:40.623576 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.623438 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g"] Nov 01 06:09:40.642684 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.642654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:40.642840 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.642729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:40.642840 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.642757 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:40.642840 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.642823 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.642804246 +0000 UTC m=+39.007102751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : configmap "kube-rbac-proxy" not found Nov 01 06:09:40.642957 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.642865 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:40.642957 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.642906 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:42.642894012 +0000 UTC m=+39.007192529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:40.706903 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.706876 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9gkn9\"" Nov 01 06:09:40.707776 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.707745 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 06:09:40.707942 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.707847 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:40.743265 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.743232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjgw\" (UniqueName: \"kubernetes.io/projected/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a-kube-api-access-fmjgw\") pod \"migrator-66df5d496b-qq59g\" (UID: \"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:40.844863 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.844809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:40.845028 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.844900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:40.845028 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.844947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:40.845028 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.844968 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845033 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.845018514 +0000 UTC m=+45.209317018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845052 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845090 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845098 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845109 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.845092857 +0000 UTC m=+45.209391370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845110 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845128 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.845117908 +0000 UTC m=+45.209416416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.845052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:40.845179 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845154 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.845141514 +0000 UTC m=+45.209440019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:40.845644 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.845194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjgw\" (UniqueName: \"kubernetes.io/projected/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a-kube-api-access-fmjgw\") pod \"migrator-66df5d496b-qq59g\" (UID: \"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:40.845644 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.845225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:40.845644 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.845250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:40.845644 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845372 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.845361197 +0000 UTC m=+45.209659704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:40.845644 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845393 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:40.845644 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.845421 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.845412368 +0000 UTC m=+45.209710870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:40.905660 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.905592 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjgw\" (UniqueName: \"kubernetes.io/projected/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a-kube-api-access-fmjgw\") pod \"migrator-66df5d496b-qq59g\" (UID: \"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:40.946033 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.946000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:40.946534 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:40.946303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:40.946534 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.946382 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:40.946534 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.946404 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:40.946534 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.946449 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:40.946534 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.946456 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.946438208 +0000 UTC m=+45.310736711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:40.946534 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:40.946510 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:48.94649566 +0000 UTC m=+45.310794167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:41.200343 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:41.200247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:41.339419 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:41.338830 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" event={"ID":"0e300938-7fc7-451b-9ca5-899929edea57","Type":"ContainerStarted","Data":"d3f15f93d124a224bdd8102a574f6e84fe91f7ce6a886dbbe8edeff756cbb049"} Nov 01 06:09:41.450732 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:41.450646 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" podStartSLOduration=385.903179016 podStartE2EDuration="6m32.45062981s" totalImagesPullingTime="6.547450794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:41.449664366 +0000 UTC m=+37.813962891" watchObservedRunningTime="2026-11-01 06:09:41.45062981 +0000 UTC m=+37.814928335" Nov 01 06:09:41.603160 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:41.603132 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g"] Nov 01 06:09:41.606191 ip-10-0-130-54 kubenswrapper[2575]: W1101 06:09:41.606162 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a138fd_e92b_4ac3_97ad_c6ad969cc43a.slice/crio-94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed WatchSource:0}: Error finding container 94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed: Status 404 returned error can't find the container with id 94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed Nov 01 06:09:42.344155 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:42.344118 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kx8mt" event={"ID":"62a9bf0c-6122-4683-80bf-43370ac319a4","Type":"ContainerStarted","Data":"33957e089aa3dd5519bab65327d1960b6474fdfb042455bedb85ce228f0885ef"} Nov 01 06:09:42.344631 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:42.344333 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:42.345538 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:42.345504 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" event={"ID":"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a","Type":"ContainerStarted","Data":"94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed"} Nov 01 06:09:42.456686 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:42.456633 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kx8mt" podStartSLOduration=35.53579045 podStartE2EDuration="38.45661573s" totalImagesPullingTime="2.92082528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:09:42.455733392 +0000 UTC m=+38.820031938" watchObservedRunningTime="2026-11-01 06:09:42.45661573 +0000 UTC m=+38.820914253" Nov 01 06:09:42.665948 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:42.665907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:42.666140 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:42.666002 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:42.666140 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:42.666010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:42.666140 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:42.666064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.666047865 +0000 UTC m=+43.030346387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : configmap "kube-rbac-proxy" not found Nov 01 06:09:42.666330 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:42.666151 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:42.666330 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:42.666185 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.666177113 +0000 UTC m=+43.030475626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:43.417104 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:43.417067 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:43.417558 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:43.417129 2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:43.417558 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:43.417505 2575 scope.go:122] "RemoveContainer" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" Nov 01 06:09:43.417709 ip-10-0-130-54 kubenswrapper[2575]: E1101 06:09:43.417688 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fqwwj_openshift-console-operator(1b4526c0-30f7-4558-b9e6-4a949e19e8fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" Nov 01 06:09:44.353423 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:44.353388 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rj4mc" event={"ID":"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a","Type":"ContainerStarted","Data":"f43582a8d9129c931ba98d6b8caad717cc9e730f45315e4500e9814df69284e2"} Nov 01 06:09:44.354840 ip-10-0-130-54 kubenswrapper[2575]: I1101 06:09:44.354815 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" event={"ID":"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a","Type":"ContainerStarted","Data":"a49aed90ff6e75af1319590a0bdf06879636542ce164f2064bdec5d5bd22c950"} Nov 01 06:09:44.372359 ip-10-0-130-54 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 06:09:44.379793 ip-10-0-130-54 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 06:09:44.379991 ip-10-0-130-54 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 06:09:44.380175 ip-10-0-130-54 systemd[1]: kubelet.service: Consumed 2.847s CPU time. Nov 01 06:09:44.409713 ip-10-0-130-54 systemd[1]: Starting Kubernetes Kubelet... Nov 01 06:09:44.530616 ip-10-0-130-54 kubenswrapper[4914]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:44.530616 ip-10-0-130-54 kubenswrapper[4914]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 06:09:44.530616 ip-10-0-130-54 kubenswrapper[4914]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:44.530616 ip-10-0-130-54 kubenswrapper[4914]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 06:09:44.532754 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.532663 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.532911 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.532900 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.532982 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.532975 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.533043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533037 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.533096 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533090 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.533155 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533149 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.533209 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533203 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.533299 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533257 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.533299 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533288 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.533299 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533294 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.533299 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533298 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.533299 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533301 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533337 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533340 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533357 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533360 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533363 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533472 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533477 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533480 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533483 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533486 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533488 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533491 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.533486 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533494 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533497 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533499 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533502 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533505 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533507 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533510 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533512 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533515 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533518 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533520 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533523 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533526 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533529 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533531 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533533 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533536 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533539 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533541 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.533788 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533549 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533552 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533555 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533558 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533560 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533563 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533565 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533567 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533570 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533572 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533575 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533577 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533581 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533583 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533586 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533590 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533595 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533598 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533603 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.534246 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533606 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533609 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533611 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533614 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533616 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533619 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533621 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533624 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533626 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533629 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533631 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533634 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533636 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533639 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533641 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533645 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533649 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533652 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533655 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.534699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533657 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533659 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533662 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533664 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533667 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533669 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533672 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533675 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533679 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533681 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533684 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533686 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533690 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533694 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533697 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533700 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533702 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533705 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533707 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533709 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.535128 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533712 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533714 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533717 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533720 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.533722 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534475 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534483 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534486 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534489 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534492 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534494 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534497 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534500 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534503 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534505 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534509 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534511 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534514 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534517 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534519 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.535645 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534521 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534524 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534526 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534529 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534531 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534534 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534536 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534538 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534541 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534543 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534545 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534548 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534550 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534552 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534555 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534557 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534559 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534562 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534564 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534566 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.536104 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534568 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534571 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534574 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534577 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534580 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534584 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534587 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534591 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534594 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534597 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534599 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534602 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534604 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534606 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534609 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534611 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534613 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534615 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534618 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.536571 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534620 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534622 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534625 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534627 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534629 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534632 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534634 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534636 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534638 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534641 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534643 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534645 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534648 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534650 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534653 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534656 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534659 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534661 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534665 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534667 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.537042 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534670 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534672 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534675 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534677 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534680 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534682 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534685 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534687 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534690 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534692 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534694 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534697 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534699 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534702 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534704 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534707 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534709 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534712 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534714 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534716 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.537531 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534719 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534721 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534724 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534726 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534728 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534731 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534734 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534738 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534743 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534745 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534748 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.534750 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535190 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535196 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535201 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535204 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535206 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535209 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535211 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.538012 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535214 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535216 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535218 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535221 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535223 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535226 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535228 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535231 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535233 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535236 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535238 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535241 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535243 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535246 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535248 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535250 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535253 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535255 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535258 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535260 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.538457 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535263 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535265 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535268 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535287 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535291 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535295 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535299 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535303 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535307 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535311 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535314 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535317 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535319 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535322 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535324 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535327 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535329 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535331 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535334 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.539015 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535336 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535339 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535341 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535343 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535346 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535348 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535350 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535353 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535355 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535357 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535360 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535363 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535366 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535368 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535374 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535377 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535379 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535382 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535384 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535387 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.539805 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535389 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535392 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535395 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535399 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535401 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535404 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535406 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535408 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535411 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535414 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535416 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535418 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535421 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535423 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535425 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535427 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535430 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535432 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535434 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535437 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.540657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535439 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535442 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535444 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535447 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535450 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535452 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535456 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535458 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535461 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535463 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535467 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535471 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535474 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535476 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535479 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535482 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535486 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535488 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535490 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.541536 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.535493 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535581 4914 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535588 4914 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535593 4914 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535597 4914 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535602 4914 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535605 4914 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535609 4914 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535612 4914 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535615 4914 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535618 4914 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535621 4914 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535624 4914 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535628 4914 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535631 4914 flags.go:64] FLAG: --cgroup-root="" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535633 4914 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535636 4914 flags.go:64] FLAG: --client-ca-file="" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535639 4914 flags.go:64] FLAG: --cloud-provider="external" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535642 4914 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535646 4914 flags.go:64] FLAG: --cluster-domain="" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535650 4914 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535653 4914 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535656 4914 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535659 4914 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 06:09:44.542386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535663 4914 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535666 4914 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535669 4914 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535672 4914 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535674 4914 flags.go:64] FLAG: --contention-profiling="false" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535678 4914 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535681 4914 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535683 4914 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535686 4914 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535690 4914 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535693 4914 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535696 4914 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535698 4914 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535701 4914 flags.go:64] FLAG: --enable-server="true" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535704 4914 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535707 4914 flags.go:64] FLAG: --event-burst="100" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535710 4914 flags.go:64] FLAG: --event-qps="50" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535713 4914 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535715 4914 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535718 4914 flags.go:64] FLAG: --eviction-hard="" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535722 4914 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535726 4914 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535728 4914 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535731 4914 flags.go:64] FLAG: --eviction-soft="" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535734 4914 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 06:09:44.543432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535737 4914 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535740 4914 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535743 4914 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535746 4914 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535749 4914 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535752 4914 flags.go:64] FLAG: --feature-gates="" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535755 4914 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535758 4914 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535761 4914 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535764 4914 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535767 4914 flags.go:64] FLAG: --healthz-port="10248" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535770 4914 flags.go:64] FLAG: --help="false" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535772 4914 flags.go:64] FLAG: --hostname-override="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535776 4914 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535778 4914 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535781 4914 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535784 4914 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535787 4914 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535790 4914 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535793 4914 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535796 4914 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535798 4914 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535801 4914 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 06:09:44.544534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535804 4914 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535807 4914 flags.go:64] FLAG: --kube-reserved="" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535809 4914 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535812 4914 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535814 4914 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535817 4914 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535820 4914 flags.go:64] FLAG: --lock-file="" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535822 4914 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535825 4914 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535828 4914 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535833 4914 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535836 4914 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535839 4914 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535842 4914 flags.go:64] FLAG: --logging-format="text" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535844 4914 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535847 4914 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535850 4914 flags.go:64] FLAG: --manifest-url="" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535853 4914 flags.go:64] FLAG: --manifest-url-header="" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535857 4914 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535860 4914 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535864 4914 flags.go:64] FLAG: --max-pods="110" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535866 4914 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535869 4914 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535872 4914 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535875 4914 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 06:09:44.545510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535878 4914 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535880 4914 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535883 4914 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535890 4914 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535893 4914 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535895 4914 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535898 4914 flags.go:64] FLAG: --pod-cidr="" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535901 4914 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535904 4914 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535906 4914 flags.go:64] FLAG: --pods-per-core="0" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535909 4914 flags.go:64] FLAG: --port="10250" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535912 4914 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535914 4914 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0cc1c1bc772332679" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535917 4914 flags.go:64] FLAG: --qos-reserved="" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535920 4914 flags.go:64] FLAG: --read-only-port="10255" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535923 4914 flags.go:64] FLAG: --register-node="true" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535925 4914 flags.go:64] FLAG: --register-with-taints="" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535929 4914 flags.go:64] FLAG: --registry-burst="10" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535933 4914 flags.go:64] FLAG: --registry-qps="5" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535936 4914 flags.go:64] FLAG: --reserved-cpus="" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535938 4914 flags.go:64] FLAG: --reserved-memory="" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535942 4914 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535945 4914 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535947 4914 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535950 4914 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535953 4914 flags.go:64] FLAG: --runonce="false" Nov 01 06:09:44.546531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535955 4914 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535959 4914 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535962 4914 flags.go:64] FLAG: --seccomp-default="false" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535964 4914 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535967 4914 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535970 4914 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535973 4914 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535976 4914 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535979 4914 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535982 4914 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535984 4914 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535987 4914 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535989 4914 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535992 4914 flags.go:64] FLAG: --system-cgroups="" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.535995 4914 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536000 4914 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536002 4914 flags.go:64] FLAG: --tls-cert-file="" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536005 4914 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536009 4914 flags.go:64] FLAG: --tls-min-version="" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536012 4914 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536014 4914 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536017 4914 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536019 4914 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536022 4914 flags.go:64] FLAG: --v="2" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536028 4914 flags.go:64] FLAG: --version="false" Nov 01 06:09:44.547513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536032 4914 flags.go:64] FLAG: --vmodule="" Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536035 4914 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536038 4914 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536173 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536176 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536180 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536183 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536185 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536187 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536190 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536192 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536195 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536197 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536203 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536207 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536210 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536212 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536215 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536217 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536220 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.548447 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536222 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536224 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536226 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536229 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536231 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536234 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536236 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536238 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536240 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536243 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536245 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536249 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536251 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536253 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536256 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536258 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536261 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536264 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536268 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.548939 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536286 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536290 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536294 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536297 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536301 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536305 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536308 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536313 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536316 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536318 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536321 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536323 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536325 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536328 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536331 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536333 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536335 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536338 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536340 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536342 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.549409 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536345 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536347 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536349 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536352 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536355 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536358 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536360 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536363 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536365 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536368 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536370 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536373 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536375 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536378 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536380 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536382 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536385 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536387 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536389 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536393 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.549865 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536396 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536398 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536401 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536404 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536406 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536408 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536412 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536415 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536418 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536421 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536423 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536426 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536428 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536430 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536432 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536435 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536439 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536441 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536444 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.550597 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536446 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536448 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536451 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536453 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536456 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536458 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536460 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536463 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536465 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536468 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536470 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.536494 4914 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536530 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536535 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536538 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536541 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.551254 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536543 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536546 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536548 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536551 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536553 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536555 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536558 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536560 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536563 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536565 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536568 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536570 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536572 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536575 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536578 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536581 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536584 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536586 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536588 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536591 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.551688 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536593 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536595 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536598 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536600 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536602 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536605 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536607 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536609 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536613 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536615 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536618 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536621 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536623 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536625 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536629 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536633 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536636 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536639 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536641 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.552170 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536643 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536646 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536648 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536650 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536653 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536655 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536657 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536660 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536662 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536664 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536666 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536669 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536671 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536673 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536676 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536678 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536680 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536683 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536685 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536688 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.552634 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536690 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536693 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536696 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536698 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536700 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536703 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536706 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536709 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536711 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536713 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536716 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536718 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536721 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536723 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536725 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536727 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536730 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536733 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536735 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536737 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.553127 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536740 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536742 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536744 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536747 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536751 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536754 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536757 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536759 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536761 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536764 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536766 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536768 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536770 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536774 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536777 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536780 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536782 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536784 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536787 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.553598 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536789 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536791 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536801 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.536804 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.541082 4914 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.541101 4914 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.541126 4914 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.541133 4914 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541212 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541221 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541226 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541230 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541235 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541239 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541243 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541247 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541251 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541256 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.554043 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541261 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541265 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541269 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541292 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541297 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541301 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541305 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541309 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541314 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541318 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541324 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541331 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541336 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541340 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541344 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541348 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541352 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541357 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541360 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.554474 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541366 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541370 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541375 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541379 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541382 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541386 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541390 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541394 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541399 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541403 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541407 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541411 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541415 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541419 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541423 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541426 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541430 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541434 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541439 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541443 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.554893 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541446 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541450 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541456 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541461 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541465 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541469 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541473 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541477 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541481 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541485 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541489 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541492 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541496 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541501 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541505 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541510 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541514 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541517 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541521 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541525 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.555375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541529 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541534 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541537 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541541 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541545 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541548 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541552 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541556 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541560 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541565 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541569 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541573 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541577 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541581 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541585 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541589 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541593 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541597 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541600 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.555809 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541604 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541609 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541613 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541616 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541620 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541624 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541629 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541634 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541638 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541642 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541646 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541650 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541656 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541662 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541667 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541671 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541675 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.556268 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541679 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.541720 4914 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541923 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541933 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541937 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541941 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541945 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541949 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541953 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541958 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541962 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541966 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541970 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541973 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541978 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541982 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.556699 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.541986 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542000 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542004 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542008 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542012 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542016 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542020 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542024 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542028 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542032 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542036 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542039 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542044 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542048 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542051 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542055 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542059 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542063 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542067 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.557175 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542071 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542075 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542079 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542082 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542086 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542090 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542094 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542097 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542101 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542104 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542108 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542112 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542116 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542120 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542123 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542128 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542132 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542136 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542139 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542143 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.557629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542147 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542151 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542155 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542161 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542167 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542175 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542179 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542183 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542188 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542192 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542195 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542200 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542203 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542207 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542211 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542215 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542219 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542223 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542227 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.558084 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542230 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542235 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542239 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542242 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542247 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542251 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542255 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542259 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542263 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542267 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542286 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542291 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542294 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542298 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542302 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542307 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542310 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542314 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542322 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542327 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.558587 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542331 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542335 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542339 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542343 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542347 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542351 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542355 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542359 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542363 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542367 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542371 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542375 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542379 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542383 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.542419 4914 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542473 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 06:09:44.559178 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542480 4914 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542484 4914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542488 4914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542492 4914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542496 4914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542500 4914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542504 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542508 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542512 4914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542515 4914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542519 4914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542523 4914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542528 4914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542532 4914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542536 4914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542542 4914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542546 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542550 4914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542554 4914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 06:09:44.559580 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542558 4914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542562 4914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542566 4914 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542570 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542574 4914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542578 4914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542582 4914 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542586 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542590 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542594 4914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542598 4914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542604 4914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542608 4914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542611 4914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542615 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542619 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542623 4914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542627 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542631 4914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542635 4914 feature_gate.go:386] unrecognized feature gate: Example Nov 01 06:09:44.560010 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542639 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542643 4914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542647 4914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542651 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542655 4914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542659 4914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542667 4914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542671 4914 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542676 4914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542682 4914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542688 4914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542692 4914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542696 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542700 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542704 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542709 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542713 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542717 4914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542720 4914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 06:09:44.560472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542724 4914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542728 4914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542733 4914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542737 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542743 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542747 4914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542752 4914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542756 4914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542760 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542764 4914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542768 4914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542772 4914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542777 4914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542781 4914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542785 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542789 4914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542793 4914 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542797 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542801 4914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542805 4914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 06:09:44.560958 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542809 4914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542815 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542819 4914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542822 4914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542828 4914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542833 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542837 4914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542841 4914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542845 4914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542848 4914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542852 4914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542856 4914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542860 4914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542865 4914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542871 4914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542876 4914 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542883 4914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542887 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542891 4914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 06:09:44.561442 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542895 4914 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542899 4914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542905 4914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542908 4914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542912 4914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542916 4914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542920 4914 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:44.542924 4914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.542996 4914 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.544424 4914 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.544494 4914 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.544763 4914 server.go:1014] "Starting client certificate rotation" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.544914 4914 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.544956 4914 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 06:04:04 +0000 UTC" deadline="2028-07-20 11:18:47.354233046 +0000 UTC" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.544981 4914 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15053h9m2.809253542s" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.545492 4914 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:44.561875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.547122 4914 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 06:09:44.562295 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.548420 4914 log.go:25] "Validated CRI v1 runtime API" Nov 01 06:09:44.562295 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.550627 4914 log.go:25] "Validated CRI v1 image API" Nov 01 06:09:44.562295 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.551741 4914 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 06:09:44.562295 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.553463 4914 fs.go:135] Filesystem UUIDs: map[1fef74f8-81ea-4e0f-a0ed-c0c7602b6ae8:/dev/nvme0n1p3 3cc278be-9e03-490b-aeef-313c90ff8c30:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 01 06:09:44.562420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.553479 4914 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1050f3edd61abc487d1ca06a753631ea844d8800e5c6b1ef0590eca20622e9c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1050f3edd61abc487d1ca06a753631ea844d8800e5c6b1ef0590eca20622e9c1/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/822e37645d7dc97d8005d50d82b88d34a05a5c50d9703871d565a02c20deb59b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/822e37645d7dc97d8005d50d82b88d34a05a5c50d9703871d565a02c20deb59b/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02d7b7a1-c649-45d0-9c18-14859b30ba56/volumes/kubernetes.io~projected/kube-api-access-z8nmn:{mountpoint:/var/lib/kubelet/pods/02d7b7a1-c649-45d0-9c18-14859b30ba56/volumes/kubernetes.io~projected/kube-api-access-z8nmn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0846b5f6-3f41-45a7-b3c4-0402f250d3c8/volumes/kubernetes.io~projected/kube-api-access-m9fms:{mountpoint:/var/lib/kubelet/pods/0846b5f6-3f41-45a7-b3c4-0402f250d3c8/volumes/kubernetes.io~projected/kube-api-access-m9fms major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e300938-7fc7-451b-9ca5-899929edea57/volumes/kubernetes.io~projected/kube-api-access-b42fs:{mountpoint:/var/lib/kubelet/pods/0e300938-7fc7-451b-9ca5-899929edea57/volumes/kubernetes.io~projected/kube-api-access-b42fs major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14525438-a3fb-4d7f-bd93-de2468852713/volumes/kubernetes.io~projected/kube-api-access-kdgvz:{mountpoint:/var/lib/kubelet/pods/14525438-a3fb-4d7f-bd93-de2468852713/volumes/kubernetes.io~projected/kube-api-access-kdgvz major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14525438-a3fb-4d7f-bd93-de2468852713/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/14525438-a3fb-4d7f-bd93-de2468852713/volumes/kubernetes.io~secret/cert major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b4526c0-30f7-4558-b9e6-4a949e19e8fb/volumes/kubernetes.io~projected/kube-api-access-wfxck:{mountpoint:/var/lib/kubelet/pods/1b4526c0-30f7-4558-b9e6-4a949e19e8fb/volumes/kubernetes.io~projected/kube-api-access-wfxck major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b4526c0-30f7-4558-b9e6-4a949e19e8fb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1b4526c0-30f7-4558-b9e6-4a949e19e8fb/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volumes/kubernetes.io~projected/kube-api-access-x8nwm:{mountpoint:/var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volumes/kubernetes.io~projected/kube-api-access-x8nwm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fe4277a-6d12-4b3c-aaef-1abd09159ac6/volumes/kubernetes.io~projected/kube-api-access-kssnx:{mountpoint:/var/lib/kubelet/pods/4fe4277a-6d12-4b3c-aaef-1abd09159ac6/volumes/kubernetes.io~projected/kube-api-access-kssnx major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55368581-d897-48e9-bd96-95d33eba4426/volumes/kubernetes.io~projected/kube-api-access-snstj:{mountpoint:/var/lib/kubelet/pods/55368581-d897-48e9-bd96-95d33eba4426/volumes/kubernetes.io~projected/kube-api-access-snstj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a9bf0c-6122-4683-80bf-43370ac319a4/volumes/kubernetes.io~projected/kube-api-access-9qj2h:{mountpoint:/var/lib/kubelet/pods/62a9bf0c-6122-4683-80bf-43370ac319a4/volumes/kubernetes.io~projected/kube-api-access-9qj2h major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6481f3ae-016a-4640-ac15-03e11f8963c6/volumes/kubernetes.io~projected/kube-api-access-krbm9:{mountpoint:/var/lib/kubelet/pods/6481f3ae-016a-4640-ac15-03e11f8963c6/volumes/kubernetes.io~projected/kube-api-access-krbm9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c66683c-48af-4ccd-9179-f61a555c3055/volumes/kubernetes.io~projected/kube-api-access-9g9xp:{mountpoint:/var/lib/kubelet/pods/6c66683c-48af-4ccd-9179-f61a555c3055/volumes/kubernetes.io~projected/kube-api-access-9g9xp major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c66683c-48af-4ccd-9179-f61a555c3055/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6c66683c-48af-4ccd-9179-f61a555c3055/volumes/kubernetes.io~secret/serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e482476-c522-476f-ad22-ab83d135404c/volumes/kubernetes.io~projected/kube-api-access-4v5zp:{mountpoint:/var/lib/kubelet/pods/6e482476-c522-476f-ad22-ab83d135404c/volumes/kubernetes.io~projected/kube-api-access-4v5zp major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c096fb-88d1-4f4c-813d-b2095302bf8c/volumes/kubernetes.io~projected/kube-api-access-x47jb:{mountpoint:/var/lib/kubelet/pods/78c096fb-88d1-4f4c-813d-b2095302bf8c/volumes/kubernetes.io~projected/kube-api-access-x47jb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78c096fb-88d1-4f4c-813d-b2095302bf8c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/78c096fb-88d1-4f4c-813d-b2095302bf8c/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fe371b4-16fb-42ce-bdd7-5e8116bcc506/volumes/kubernetes.io~projected/kube-api-access-bmgwv:{mountpoint:/var/lib/kubelet/pods/7fe371b4-16fb-42ce-bdd7-5e8116bcc506/volumes/kubernetes.io~projected/kube-api-access-bmgwv major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8079984d-980a-4287-826e-91194cf07e9e/volumes/kubernetes.io~projected/kube-api-access-l5gkc:{mountpoint:/var/lib/kubelet/pods/8079984d-980a-4287-826e-91194cf07e9e/volumes/kubernetes.io~projected/kube-api-access-l5gkc major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c8da36-c064-46e9-9712-b36bde260e3d/volumes/kubernetes.io~projected/kube-api-access-sdxns:{mountpoint:/var/lib/kubelet/pods/80c8da36-c064-46e9-9712-b36bde260e3d/volumes/kubernetes.io~projected/kube-api-access-sdxns major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c8da36-c064-46e9-9712-b36bde260e3d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/80c8da36-c064-46e9-9712-b36bde260e3d/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~projected/kube-api-access-hl2x6:{mountpoint:/var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~projected/kube-api-access-hl2x6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa19977-5c71-40c4-acb8-29e0d569efb4/volumes/kubernetes.io~projected/kube-api-access-qthh4:{mountpoint:/var/lib/kubelet/pods/8fa19977-5c71-40c4-acb8-29e0d569efb4/volumes/kubernetes.io~projected/kube-api-access-qthh4 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99cae588-1b8c-4972-bda1-781a3edb687a/volumes/kubernetes.io~projected/kube-api-access-hnrsr:{mountpoint:/var/lib/kubelet/pods/99cae588-1b8c-4972-bda1-781a3edb687a/volumes/kubernetes.io~projected/kube-api-access-hnrsr major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~projected/kube-api-access-2zt8l:{mountpoint:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~projected/kube-api-access-2zt8l major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2b76995-eb22-418d-9665-bb729b6e88c4/volumes/kubernetes.io~projected/kube-api-access-2mh9f:{mountpoint:/var/lib/kubelet/pods/b2b76995-eb22-418d-9665-bb729b6e88c4/volumes/kubernetes.io~projected/kube-api-access-2mh9f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a/volumes/kubernetes.io~projected/kube-api-access-fmjgw:{mountpoint:/var/lib/kubelet/pods/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a/volumes/kubernetes.io~projected/kube-api-access-fmjgw major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1/volumes/kubernetes.io~projected/kube-api-access-5dtbp:{mountpoint:/var/lib/kubelet/pods/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1/volumes/kubernetes.io~projected/kube-api-access-5dtbp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~projected/kube-api-access-l68vq:{mountpoint:/var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~projected/kube-api-access-l68vq major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfef5efe-9658-4dbe-a3b5-cb484fec7962/volumes/kubernetes.io~projected/kube-api-access-8d8vr:{mountpoint:/var/lib/kubelet/pods/cfef5efe-9658-4dbe-a3b5-cb484fec7962/volumes/kubernetes.io~projected/kube-api-access-8d8vr major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7014b11-e3e2-448c-b1bc-08f298821e3b/volumes/kubernetes.io~projected/kube-api-access-9fdtv:{mountpoint:/var/lib/kubelet/pods/d7014b11-e3e2-448c-b1bc-08f298821e3b/volumes/kubernetes.io~projected/kube-api-access-9fdtv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd901c17-d465-4a03-b320-73be6ff3d0ce/volumes/kubernetes.io~projected/kube-api-access-p9rfc:{mountpoint:/var/lib/kubelet/pods/dd901c17-d465-4a03-b320-73be6ff3d0ce/volumes/kubernetes.io~projected/kube-api-access-p9rfc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e862b42b-a7df-440b-9170-89c0de89c451/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e862b42b-a7df-440b-9170-89c0de89c451/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~projected/kube-api-access-qdrqw:{mountpoint:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~projected/kube-api-access-qdrqw major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:222 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/59fc167e1cbc05feea4698ae407513b06f31d44e816bcb8991b4d1bd773e5285/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/38b5957bd6ce9973441886a3afb1e36cb0752a4d172a428e27f61f036fa3b6d4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/bb9d1c2b75f3a67f36de4e5b23c11ab405162d872e090286331b1d8ee20025c0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/6e125288e255273879c618a3b9d32e4fbb9ce4a79cbace5a21bbb9388898e3cb/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9df9d4962402496ab63ff320551b44573c42cb7616be7c7a693f487e7c37a7c1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7f02fa705c005437a43a75c062365161c0fdeb1ed41cf3e3b74eefb699e0ad27/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e9ac0343ceebdb7a59a33aee35e9a190136b630d21f177c2bfbc053c2d20ff14/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/27a65823670399343d00cd97c04bb79aca4480c79ddb58f3faee79ca8b816c14/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fa70cc07eae0b02c315c0dff7c62a44d968f3e1bdfd11a40e3621196ec049171/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/3167650916b3d1245857fed387fadd107e583f3441a9a145f5a09c7d9708bd75/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/3ff4ecceaa021cea854c809ca740c69c2757db0aa5b999bafd3d58727e764d68/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/6e0933d2f612554e9b81d8b2cbcc45b1904517993dc26cc758518c3df9087e0a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/3b53cfb49939f0cbd94b0c954daa1aeaca528f89ccb61dc5094dcab17390da65/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/419d436251cfab953c02082a7839b72d68899170c7ad08b07683e2efa2c98542/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/00ece0ed07a559d323dbb9fb2803b95f9c01f906899e612b079872cdaf79734a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fd6f7f1887a72acae8ed93068f1a2c0a0ef03d88edab16f5351b4df380853816/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b1262318a8acb68b4601efa63f16f13bb68b84ba6fa81d4285a93c1408d9bdc8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3cd75612c0709419e257fff58187c4916d001ea7e44dbc001aaab82459434d49/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ddd4aa4540595df0817023cb5f0513ddbf82c34135b85ac045f9580a762a6217/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6528ae96339537bdc5d8c60ab7432275ac531de7b2ca07e5b9ebf2af56d00264/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/69752ae2ff3743f3770e5035eda8723330d8b875c12104ff395d03919164720a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/406e584010907fa43a9823e51e8220baff197b58a977ed2a48fa3e30aa6bab49/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fa370ab50cdd00c585a754d8a8dafc2f170676729179c4bb6be9ddc455173f7b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/fe8752728294eee1f70ec77855c3137b304b9d64f63197e7f916cbe17bbf33b8/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/84e015de35ec4270565f59929320a7a1ec966ebf932d9eaa966a766effd3c173/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1dfd33ae4a5d2f8516b72088e64b58f5891edf2bd19679e9d8007d0e44b64887/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/94dcadf9c8e3d9c42c619faf6187c074720b3e9219adfb861e159b5de150f96f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/ea4a950bc7cb3d3d60a4d1800b6e035d92b31fc2e77208d56d8a89e1559bbcb2/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/c131635ff42e96d19bb8f97d05bc47ce94fc6f7254f715b09491f90e5d46153c/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/7f0d5c24b34d14a6eb3666135c68f37cb5cc02bd4ac382608b923595891af092/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/51b223a2c42c0d1391d1a6797c16f15b8e38d92746378f53f1bb680a05a6236a/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/933771884a16a3defd914d64a0cca5fc8e43ddf084b3bef0b8902954e9cf7160/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/6381e63534e8fe7096a991ad8d0435f22ae2555f51408942dfa442d2c9012358/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/99cf0d58632f6dcf53e5ff75a53c04934d46ed8c6a9189a24e186ccc4b8e109c/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/c105ab667125968dfd00ff0b5a83eea5f89adc3c76d821a29841d8b983dd1922/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/a397c2fad9b483b2fd035a217e5ac40f81be2bf34ef206184dcfb520720126e2/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d9baff3f907c948794f456902efa23dac56d1ec157f15645a2eaf9043bc29a3e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/efd8d873f23ee60144e82ede65a419de311cab88ff05103e9246e74dd55af293/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/83104387dd0ab1ef8b66b4d3387184793ceb28e794a08f490da455d52b94b2a7/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/ba6db622c9ea357f6bef2e77774ff2e6f57f30dae5cc36798a4447e41af3f555/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/0a0ee8c43e4dc92c26c1a58e42ef477caa909eb4dc5834639378b20697e7892f/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/c236235d3c6e271724ffe8ecf40a9f9c0bceb10c2bb174c42afb9acec324b4e0/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/b0488c0fb17eb2e6c83d424e7001d3296c5ad4f951623f9e7b39524e632357e9/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/22986064a6e5ecbd8b11d9a180041a988b23c221ff380e0ef82af9e7548e8a39/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/8293b6f354b42bee9df8d01294bec837b4c1ad33880ad45042765c1c12096692/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c5f354404f622e5cb106f32df1bfdcbc04bba39e88362dcfd5c50d9be7871643/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/399df48fe153f7bb98a5e40d2c8e23a6acb39f59b42468fbe0345943fcd9d8b7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/636e29f3fdb8959a5c7d21de468e1fc633cc3b91e5fc20644ea31f795de8c484/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/15d79b858388f146973cdfcf322613815fb56bb2ab4b8b8d756f9910768caf7e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9eccfa48a4796d967b3d4184e75fd67d95e6656e5d5a4c5eec02689f7d808941/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6ec1786f493f78aa8d7401ad619d3bf21bf35fc3176a07e8ed9a614382de0d52/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/afa20f566f9104fe1dd9a69c02f4978fc46ca13362e512d2441643be1ce345b2/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/6f928d9594b7c8eceaaa8c06626e2df1a31f730827d53494bc952c96b6e4ab5d/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/a897ea6c30c1f043823472754fa5c076158edee94a8c4644f1ee818e5c145980/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7bbfeb853d2da4bdd19d4cf319df750e88892cd3e4d87957db0239a215526ddb/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/3708654c7c296f0427d393345fec5d13bf1914d18d0bd3494271c9646a3c4779/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.561471 4914 manager.go:217] Machine: {Timestamp:2026-11-01 06:09:44.561021219 +0000 UTC m=+0.113810403 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3253367 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b1a7544627809171f7155ce03414 SystemUUID:ec28b1a7-5446-2780-9171-f7155ce03414 BootID:eab2a777-557f-4ea7-8677-b597860de589 Filesystems:[{Device:/var/lib/kubelet/pods/6481f3ae-016a-4640-ac15-03e11f8963c6/volumes/kubernetes.io~projected/kube-api-access-krbm9 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5242a0f32240a0b244e575482fcaae019e61d664c534ebd0d94f97694ed84473/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b4526c0-30f7-4558-b9e6-4a949e19e8fb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78c096fb-88d1-4f4c-813d-b2095302bf8c/volumes/kubernetes.io~projected/kube-api-access-x47jb DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14525438-a3fb-4d7f-bd93-de2468852713/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14525438-a3fb-4d7f-bd93-de2468852713/volumes/kubernetes.io~projected/kube-api-access-kdgvz DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e862b42b-a7df-440b-9170-89c0de89c451/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c66683c-48af-4ccd-9179-f61a555c3055/volumes/kubernetes.io~projected/kube-api-access-9g9xp DeviceMajor:0 DeviceMinor:238 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1/volumes/kubernetes.io~projected/kube-api-access-5dtbp DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/822e37645d7dc97d8005d50d82b88d34a05a5c50d9703871d565a02c20deb59b/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b1adfc3a3accd00cc014a4cc53cb286bf64058f73650acb90b62fd31c444a11/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb9daf9d30004d75368312391ad4da886ad4b3d67deafac2d87c9f7cabe014ee/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94a35859b0d3a3f67b04d6acf6e1d6b6a5da6e299cc03754cf6516fb1c704bed/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~projected/kube-api-access-2zt8l DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~projected/kube-api-access-hl2x6 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b4526c0-30f7-4558-b9e6-4a949e19e8fb/volumes/kubernetes.io~projected/kube-api-access-wfxck DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b942d3d9db2bfedd2b8cf631fc5d2caa47bab270142cfb038e57ee6e864a801b/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce6a95e33429e8b4c938d33df0d54aeb5fd34f5a76feeb54614c8b9b1b6eb965/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa19977-5c71-40c4-acb8-29e0d569efb4/volumes/kubernetes.io~projected/kube-api-access-qthh4 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c8da36-c064-46e9-9712-b36bde260e3d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/719edd60ce5b64a0a5ce250cf5fb885f8552675edbaf7ba25b23663886ad618a/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfef5efe-9658-4dbe-a3b5-cb484fec7962/volumes/kubernetes.io~projected/kube-api-access-8d8vr DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/656a74e1cefaa7b9e6a8751158816ae47a818f48fef778deffbd49723b3be3fa/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f4c42c8a888c704e3d1ab550bad7934ad3261b03590c7986f31327815e1f9d5/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f6904050e99acc85ee9f8fefa919ba7e4880a17d66055989e745975c49cf4bd/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8079984d-980a-4287-826e-91194cf07e9e/volumes/kubernetes.io~projected/kube-api-access-l5gkc DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6e482476-c522-476f-ad22-ab83d135404c/volumes/kubernetes.io~projected/kube-api-access-4v5zp DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7fe371b4-16fb-42ce-bdd7-5e8116bcc506/volumes/kubernetes.io~projected/kube-api-access-bmgwv DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/80c8da36-c064-46e9-9712-b36bde260e3d/volumes/kubernetes.io~projected/kube-api-access-sdxns DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1050f3edd61abc487d1ca06a753631ea844d8800e5c6b1ef0590eca20622e9c1/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/304c2d301aadc9d75ca490acdd73d5fb2c70c96a716c18f8e1619ac974956870/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2b76995-eb22-418d-9665-bb729b6e88c4/volumes/kubernetes.io~projected/kube-api-access-2mh9f DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/910dbcf883a521fd01af0f07662bcc90afa544344d8f9ed0f1d06485a9ebd4ea/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~projected/kube-api-access-l68vq DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02d7b7a1-c649-45d0-9c18-14859b30ba56/volumes/kubernetes.io~projected/kube-api-access-z8nmn DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/528c65beab67290a179f062e5ae541ee8fc2fa59f482613b6630ac4d9d3905c8/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0846b5f6-3f41-45a7-b3c4-0402f250d3c8/volumes/kubernetes.io~projected/kube-api-access-m9fms DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3d30aeb24363f89dbee4fdf63bef8fcbac5eb29ba9be17af3611f6dc6e12174/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc06a215-884f-40b1-90fc-208bcc319a76/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99a09c9b1d85d43ed0639bbc61becac729a664b0db17d42bdaf5c1acbfe7849c/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4fe4277a-6d12-4b3c-aaef-1abd09159ac6/volumes/kubernetes.io~projected/kube-api-access-kssnx DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99cae588-1b8c-4972-bda1-781a3edb687a/volumes/kubernetes.io~projected/kube-api-access-hnrsr DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0e300938-7fc7-451b-9ca5-899929edea57/volumes/kubernetes.io~projected/kube-api-access-b42fs DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a/volumes/kubernetes.io~projected/kube-api-access-fmjgw DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d44d5a887b40e14c305b9eb547093ae70cf30b11dd6b7b5c7490de17b121368/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd5ccddbdf550d8ddd8982ac7abf54d2bb87511ca8a0f1727e3fca5d88da9696/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~projected/kube-api-access-qdrqw DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/55368581-d897-48e9-bd96-95d33eba4426/volumes/kubernetes.io~projected/kube-api-access-snstj DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f/volumes/kubernetes.io~projected/kube-api-access-x8nwm DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c66683c-48af-4ccd-9179-f61a555c3055/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18842e82e65335fb9136ed8e4f767b2c14e1500f6c27386d7dd17b741f5926cc/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78c096fb-88d1-4f4c-813d-b2095302bf8c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/62a9bf0c-6122-4683-80bf-43370ac319a4/volumes/kubernetes.io~projected/kube-api-access-9qj2h DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7014b11-e3e2-448c-b1bc-08f298821e3b/volumes/kubernetes.io~projected/kube-api-access-9fdtv DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/502dd1881f2104cca6940f6391bcd05ee22172338a4f8e80a630df2df27649c1/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd901c17-d465-4a03-b320-73be6ff3d0ce/volumes/kubernetes.io~projected/kube-api-access-p9rfc DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1050f3edd61abc4 MacAddress:a6:37:f7:01:b0:73 Speed:10000 Mtu:8901} {Name:18842e82e65335f MacAddress:3e:3d:17:87:49:99 Speed:10000 Mtu:8901} {Name:304c2d301aadc9d MacAddress:c6:7c:0a:55:90:51 Speed:10000 Mtu:8901} {Name:528c65beab67290 MacAddress:6e:52:85:74:a6:77 Speed:10000 Mtu:8901} {Name:656a74e1cefaa7b MacAddress:ea:5a:cf:90:02:58 Speed:10000 Mtu:8901} {Name:719edd60ce5b64a MacAddress:a2:fa:67:40:98:78 Speed:10000 Mtu:8901} {Name:822e37645d7dc97 MacAddress:4a:a5:f7:dc:5e:a3 Speed:10000 Mtu:8901} {Name:94a35859b0d3a3f MacAddress:5e:28:15:3e:26:ea Speed:10000 Mtu:8901} {Name:99a09c9b1d85d43 MacAddress:4e:78:79:c4:92:64 Speed:10000 Mtu:8901} {Name:9b1adfc3a3accd0 MacAddress:82:f7:67:0a:96:81 Speed:10000 Mtu:8901} {Name:b942d3d9db2bfed MacAddress:02:8e:60:eb:a6:db Speed:10000 Mtu:8901} {Name:bb9daf9d30004d7 MacAddress:b2:00:52:e8:bb:47 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:52:37:9c:11:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:59:4d:47:cc:66 Speed:0 Mtu:8901} {Name:cd5ccddbdf550d8 MacAddress:56:77:08:34:53:be Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:52:37:9c:11:6d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7a:80:eb:2e:39:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:32:35:26:97:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.561814 4914 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.561855 4914 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562133 4914 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562149 4914 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-54.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562286 4914 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562296 4914 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562309 4914 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562320 4914 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562388 4914 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 06:09:44.562625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562531 4914 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 06:09:44.563196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562633 4914 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 06:09:44.563196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562665 4914 kubelet.go:493] "Attempting to sync node with API server" Nov 01 06:09:44.563196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562675 4914 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 06:09:44.563196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562685 4914 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 06:09:44.563196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562693 4914 kubelet.go:405] "Adding apiserver pod source" Nov 01 06:09:44.563196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.562700 4914 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 06:09:44.563594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.563579 4914 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 06:09:44.563634 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.563596 4914 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:44.564070 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.564053 4914 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 06:09:44.566056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566044 4914 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 06:09:44.566105 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566057 4914 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 06:09:44.566150 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566142 4914 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 06:09:44.566330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566320 4914 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 06:09:44.566362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566348 4914 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 06:09:44.566432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566424 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 06:09:44.566459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566437 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 06:09:44.566459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566443 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 06:09:44.566459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566448 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 06:09:44.566459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566453 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 06:09:44.566459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566458 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566464 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566469 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566475 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566480 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566488 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566497 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566515 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 06:09:44.566582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.566520 4914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 06:09:44.569653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.569635 4914 server.go:1263] "Started kubelet" Nov 01 06:09:44.569939 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.569880 4914 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 06:09:44.570608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.570009 4914 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 06:09:44.570608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.570078 4914 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 06:09:44.570547 ip-10-0-130-54 systemd[1]: Started Kubernetes Kubelet. Nov 01 06:09:44.571907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.571879 4914 server.go:317] "Adding debug handlers to kubelet server" Nov 01 06:09:44.572700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.572393 4914 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 06:09:44.575685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575667 4914 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 06:09:44.575770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575685 4914 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 06:09:44.575770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575727 4914 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 06:09:44.575770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575741 4914 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575816 4914 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 06:04:06 +0000 UTC" deadline="2028-05-11 21:00:21.251071286 +0000 UTC" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575846 4914 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13382h50m36.675228703s" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.575871 4914 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.575882 4914 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.576488 4914 factory.go:55] Registering systemd factory Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.576506 4914 factory.go:223] Registration of the systemd container factory successfully Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.577176 4914 factory.go:153] Registering CRI-O factory Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.577192 4914 factory.go:223] Registration of the crio container factory successfully Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.577258 4914 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.577299 4914 factory.go:103] Registering Raw factory Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.577315 4914 manager.go:1199] Started watching for new ooms in manager Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.577649 4914 manager.go:319] Starting recovery of all containers Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579018 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579052 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" volumeName="kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-config" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579087 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1caa2f5d-b542-4af5-8dc4-3b75ab99c08f" volumeName="kubernetes.io/projected/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-kube-api-access-x8nwm" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579100 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe4277a-6d12-4b3c-aaef-1abd09159ac6" volumeName="kubernetes.io/empty-dir/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-tmp-dir" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579114 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6481f3ae-016a-4640-ac15-03e11f8963c6" volumeName="kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579126 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6481f3ae-016a-4640-ac15-03e11f8963c6" volumeName="kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579136 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e482476-c522-476f-ad22-ab83d135404c" volumeName="kubernetes.io/projected/6e482476-c522-476f-ad22-ab83d135404c-kube-api-access-4v5zp" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579147 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc" volumeName="kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-tuned" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579177 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579188 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2b76995-eb22-418d-9665-bb729b6e88c4" volumeName="kubernetes.io/projected/b2b76995-eb22-418d-9665-bb729b6e88c4-kube-api-access-2mh9f" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579201 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc06a215-884f-40b1-90fc-208bcc319a76" volumeName="kubernetes.io/projected/cc06a215-884f-40b1-90fc-208bcc319a76-kube-api-access-l68vq" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579213 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579225 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0846b5f6-3f41-45a7-b3c4-0402f250d3c8" volumeName="kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cni-binary-copy" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579236 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1caa2f5d-b542-4af5-8dc4-3b75ab99c08f" volumeName="kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-env-overrides" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579251 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55368581-d897-48e9-bd96-95d33eba4426" volumeName="kubernetes.io/projected/55368581-d897-48e9-bd96-95d33eba4426-kube-api-access-snstj" seLinuxMountContext="" Nov 01 06:09:44.579256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579262 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6481f3ae-016a-4640-ac15-03e11f8963c6" volumeName="kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-binary-copy" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579290 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c096fb-88d1-4f4c-813d-b2095302bf8c" volumeName="kubernetes.io/projected/78c096fb-88d1-4f4c-813d-b2095302bf8c-kube-api-access-x47jb" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579301 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579312 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579323 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" volumeName="kubernetes.io/projected/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-kube-api-access-wfxck" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579335 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1caa2f5d-b542-4af5-8dc4-3b75ab99c08f" volumeName="kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-script-lib" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579347 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe4277a-6d12-4b3c-aaef-1abd09159ac6" volumeName="kubernetes.io/configmap/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-config-volume" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579358 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e482476-c522-476f-ad22-ab83d135404c" volumeName="kubernetes.io/configmap/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-config" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579369 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579381 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c8da36-c064-46e9-9712-b36bde260e3d" volumeName="kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-trusted-ca-bundle" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579391 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579402 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579412 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2b76995-eb22-418d-9665-bb729b6e88c4" volumeName="kubernetes.io/empty-dir/b2b76995-eb22-418d-9665-bb729b6e88c4-tmp-dir" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579429 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd901c17-d465-4a03-b320-73be6ff3d0ce" volumeName="kubernetes.io/projected/dd901c17-d465-4a03-b320-73be6ff3d0ce-kube-api-access-p9rfc" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579440 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c8da36-c064-46e9-9712-b36bde260e3d" volumeName="kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-service-ca-bundle" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579452 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e862b42b-a7df-440b-9170-89c0de89c451" volumeName="kubernetes.io/configmap/e862b42b-a7df-440b-9170-89c0de89c451-konnectivity-ca" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579467 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a9bf0c-6122-4683-80bf-43370ac319a4" volumeName="kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579478 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8079984d-980a-4287-826e-91194cf07e9e" volumeName="kubernetes.io/projected/8079984d-980a-4287-826e-91194cf07e9e-kube-api-access-l5gkc" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579489 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc" volumeName="kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-tmp" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579501 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579511 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfef5efe-9658-4dbe-a3b5-cb484fec7962" volumeName="kubernetes.io/configmap/cfef5efe-9658-4dbe-a3b5-cb484fec7962-telemetry-config" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579522 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579534 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6481f3ae-016a-4640-ac15-03e11f8963c6" volumeName="kubernetes.io/projected/6481f3ae-016a-4640-ac15-03e11f8963c6-kube-api-access-krbm9" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579545 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1" volumeName="kubernetes.io/projected/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-kube-api-access-5dtbp" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579555 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14525438-a3fb-4d7f-bd93-de2468852713" volumeName="kubernetes.io/projected/14525438-a3fb-4d7f-bd93-de2468852713-kube-api-access-kdgvz" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579568 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c66683c-48af-4ccd-9179-f61a555c3055" volumeName="kubernetes.io/secret/6c66683c-48af-4ccd-9179-f61a555c3055-serving-cert" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579580 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc06a215-884f-40b1-90fc-208bcc319a76" volumeName="kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-default-certificate" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579590 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579602 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0846b5f6-3f41-45a7-b3c4-0402f250d3c8" volumeName="kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-daemon-config" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579614 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a" volumeName="kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579624 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef1e9b60-c1a2-4e84-b155-d43c5b023617" volumeName="kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579636 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02d7b7a1-c649-45d0-9c18-14859b30ba56" volumeName="kubernetes.io/projected/02d7b7a1-c649-45d0-9c18-14859b30ba56-kube-api-access-z8nmn" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579647 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e300938-7fc7-451b-9ca5-899929edea57" volumeName="kubernetes.io/projected/0e300938-7fc7-451b-9ca5-899929edea57-kube-api-access-b42fs" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579658 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" volumeName="kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-trusted-ca" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579670 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c8da36-c064-46e9-9712-b36bde260e3d" volumeName="kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-snapshots" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579681 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa19977-5c71-40c4-acb8-29e0d569efb4" volumeName="kubernetes.io/projected/8fa19977-5c71-40c4-acb8-29e0d569efb4-kube-api-access-qthh4" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579692 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc" volumeName="kubernetes.io/projected/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-kube-api-access-hl2x6" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579705 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7014b11-e3e2-448c-b1bc-08f298821e3b" volumeName="kubernetes.io/projected/d7014b11-e3e2-448c-b1bc-08f298821e3b-kube-api-access-9fdtv" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579717 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0846b5f6-3f41-45a7-b3c4-0402f250d3c8" volumeName="kubernetes.io/projected/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-kube-api-access-m9fms" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579728 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fe4277a-6d12-4b3c-aaef-1abd09159ac6" volumeName="kubernetes.io/projected/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-kube-api-access-kssnx" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579740 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55368581-d897-48e9-bd96-95d33eba4426" volumeName="kubernetes.io/configmap/55368581-d897-48e9-bd96-95d33eba4426-iptables-alerter-script" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579758 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1caa2f5d-b542-4af5-8dc4-3b75ab99c08f" volumeName="kubernetes.io/secret/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579769 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c66683c-48af-4ccd-9179-f61a555c3055" volumeName="kubernetes.io/configmap/6c66683c-48af-4ccd-9179-f61a555c3055-config" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579783 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe371b4-16fb-42ce-bdd7-5e8116bcc506" volumeName="kubernetes.io/projected/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-api-access-bmgwv" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579795 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c8da36-c064-46e9-9712-b36bde260e3d" volumeName="kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-tmp" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579806 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c8da36-c064-46e9-9712-b36bde260e3d" volumeName="kubernetes.io/secret/80c8da36-c064-46e9-9712-b36bde260e3d-serving-cert" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579817 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579830 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1" volumeName="kubernetes.io/configmap/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-serviceca" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579840 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfef5efe-9658-4dbe-a3b5-cb484fec7962" volumeName="kubernetes.io/projected/cfef5efe-9658-4dbe-a3b5-cb484fec7962-kube-api-access-8d8vr" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579852 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14525438-a3fb-4d7f-bd93-de2468852713" volumeName="kubernetes.io/secret/14525438-a3fb-4d7f-bd93-de2468852713-cert" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579864 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1caa2f5d-b542-4af5-8dc4-3b75ab99c08f" volumeName="kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-config" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579874 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c66683c-48af-4ccd-9179-f61a555c3055" volumeName="kubernetes.io/projected/6c66683c-48af-4ccd-9179-f61a555c3055-kube-api-access-9g9xp" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579898 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c096fb-88d1-4f4c-813d-b2095302bf8c" volumeName="kubernetes.io/secret/78c096fb-88d1-4f4c-813d-b2095302bf8c-serving-cert" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579911 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c8da36-c064-46e9-9712-b36bde260e3d" volumeName="kubernetes.io/projected/80c8da36-c064-46e9-9712-b36bde260e3d-kube-api-access-sdxns" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579922 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f7226f4-6e78-4172-8f28-c3225d93636f" volumeName="kubernetes.io/configmap/8f7226f4-6e78-4172-8f28-c3225d93636f-nginx-conf" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579934 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e862b42b-a7df-440b-9170-89c0de89c451" volumeName="kubernetes.io/secret/e862b42b-a7df-440b-9170-89c0de89c451-agent-certs" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579945 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" volumeName="kubernetes.io/secret/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-serving-cert" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579956 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78c096fb-88d1-4f4c-813d-b2095302bf8c" volumeName="kubernetes.io/configmap/78c096fb-88d1-4f4c-813d-b2095302bf8c-config" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579967 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fe371b4-16fb-42ce-bdd7-5e8116bcc506" volumeName="kubernetes.io/empty-dir/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-data-volume" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579978 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc06a215-884f-40b1-90fc-208bcc319a76" volumeName="kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-stats-auth" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.579989 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99cae588-1b8c-4972-bda1-781a3edb687a" volumeName="kubernetes.io/projected/99cae588-1b8c-4972-bda1-781a3edb687a-kube-api-access-hnrsr" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.580000 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" volumeName="kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.580012 4914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a138fd-e92b-4ac3-97ad-c6ad969cc43a" volumeName="kubernetes.io/projected/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a-kube-api-access-fmjgw" seLinuxMountContext="" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.580024 4914 reconstruct.go:97] "Volume reconstruction finished" Nov 01 06:09:44.580451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.580031 4914 reconciler.go:29] "Reconciler: start to sync state" Nov 01 06:09:44.591591 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.591566 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/1.log" Nov 01 06:09:44.591764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.591743 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/1.log" Nov 01 06:09:44.607261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.607204 4914 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 06:09:44.608517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.608496 4914 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 06:09:44.608517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.608518 4914 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 06:09:44.608647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.608536 4914 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 06:09:44.608647 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.608566 4914 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 06:09:44.618717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.618702 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/1.log" Nov 01 06:09:44.618781 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.618725 4914 generic.go:356] "Generic (PLEG): container finished" podID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" exitCode=255 Nov 01 06:09:44.634500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.634471 4914 generic.go:356] "Generic (PLEG): container finished" podID="5c62962b404022a5a6674917fdd042e4" containerID="b94189fd9a901225b1348266350d53741871a369bcc4f700cb9ef390c68434f7" exitCode=0 Nov 01 06:09:44.648408 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.648381 4914 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="449b5dbcfc0e08a6e52dacbfd45665d0bb1498d213e3d6f035c2881ce276c82a" exitCode=0 Nov 01 06:09:44.648496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.648411 4914 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="f3d79c6b4980da5f6dd243b4c93883f0d6c7aa550e968d4ba6c2408c784189f6" exitCode=0 Nov 01 06:09:44.648496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.648421 4914 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="7035e066194e0cf91e13571bf4ca725a08f410d0e4040b11cd7d41d896121039" exitCode=0 Nov 01 06:09:44.648496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.648431 4914 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="50726883e5fafcccabbe487c2eda70d4f6af1d89d4f28d22b086c505f5cb542d" exitCode=0 Nov 01 06:09:44.648496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.648441 4914 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="da099f88c0bd63b0341c99fd40bef1a7f0c56641aeef481aa79350ccae494ea8" exitCode=0 Nov 01 06:09:44.648496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.648451 4914 generic.go:356] "Generic (PLEG): container finished" podID="6481f3ae-016a-4640-ac15-03e11f8963c6" containerID="7f3ea5569716886b433acb9f4a30d9aab3e0412cb939cb6a01bb0d941316c7e7" exitCode=0 Nov 01 06:09:44.662190 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.662171 4914 manager.go:324] Recovery completed Nov 01 06:09:44.676000 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.675974 4914 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:44.678844 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.678831 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.679941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.679924 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.680006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.679952 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.680006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.679962 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.682115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682102 4914 cpu_manager.go:226] "Starting" policy="none" Nov 01 06:09:44.682115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682113 4914 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 06:09:44.682231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682119 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.682231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682129 4914 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 06:09:44.682305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682243 4914 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 06:09:44.682305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682252 4914 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 06:09:44.682305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682296 4914 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 06:09:44.682305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682301 4914 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 06:09:44.682305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682306 4914 policy_none.go:50] "Start" Nov 01 06:09:44.682432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682315 4914 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 06:09:44.682432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682324 4914 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 06:09:44.682500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682439 4914 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 06:09:44.682500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682446 4914 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 06:09:44.682500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682451 4914 policy_none.go:44] "Start" Nov 01 06:09:44.682923 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682907 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.682993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682934 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.682993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.682945 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.686679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.686655 4914 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 06:09:44.689253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689241 4914 manager.go:341] "Starting Device Plugin manager" Nov 01 06:09:44.689350 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.689264 4914 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 06:09:44.689350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689289 4914 server.go:87] "Starting device plugin registration server" Nov 01 06:09:44.689549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689536 4914 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 06:09:44.689591 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689550 4914 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 06:09:44.689674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689661 4914 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 06:09:44.689777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689747 4914 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 06:09:44.689777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689781 4914 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 06:09:44.689899 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689788 4914 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 06:09:44.689899 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.689831 4914 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T06:09:44.68977182Z","UUID":"b441b0f1-df0b-4b4b-bece-386eee8a9d92","Handler":null,"Name":"","Endpoint":""} Nov 01 06:09:44.691358 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.691335 4914 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 06:09:44.691444 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.691383 4914 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" Nov 01 06:09:44.691742 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.691729 4914 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 06:09:44.691807 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.691750 4914 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 06:09:44.709711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.709289 4914 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal"] Nov 01 06:09:44.709792 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.709781 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.710572 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.710557 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.710659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.710586 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.710659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.710599 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.710898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.710884 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.711007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.710994 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.711661 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711643 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.711732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711665 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.711732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711674 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.711732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711674 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.711732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711693 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.711732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711702 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.711886 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.711806 4914 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.711914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711884 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"2b0505d995be0782193889599bca279d","Type":"ContainerStarted","Data":"337f7111939c55108ec52218c7303e862c19ab651632fb4f561796ba1905b28a"} Nov 01 06:09:44.711958 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711948 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"5c62962b404022a5a6674917fdd042e4","Type":"ContainerStarted","Data":"e35f4ce991f42b378ae2853f3be2935ce1c3e797090f60f4611592578a45c993"} Nov 01 06:09:44.711985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711963 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"5c62962b404022a5a6674917fdd042e4","Type":"ContainerDied","Data":"b94189fd9a901225b1348266350d53741871a369bcc4f700cb9ef390c68434f7"} Nov 01 06:09:44.711985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711975 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" event={"ID":"5c62962b404022a5a6674917fdd042e4","Type":"ContainerStarted","Data":"a6cd699cc124b7262e378781a10ef6adc293eccb1171f6bf4014da76fa4cb6dd"} Nov 01 06:09:44.712039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.711997 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.712723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.712706 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.712785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.712731 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.712785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.712741 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.712887 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.712873 4914 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-54.us-west-1.compute.internal\" not found" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.768787 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.768759 4914 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:44.768886 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.768844 4914 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:44.768886 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.768859 4914 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 06:09:44.769377 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.769364 4914 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:44.777466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.777455 4914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.790600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.790586 4914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 06:09:44.791391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.791378 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 01 06:09:44.791460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.791397 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 01 06:09:44.791460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.791409 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 01 06:09:44.791460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.791452 4914 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.819487 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.819464 4914 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 06:09:44.860055 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.860004 4914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:44.860118 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.860053 4914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.860118 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.860068 4914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.869204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.869183 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b0505d995be0782193889599bca279d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"2b0505d995be0782193889599bca279d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.869244 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.869206 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.869298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.869239 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.911653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.911635 4914 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.911726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.911680 4914 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.912004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.911980 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeNotReady" Nov 01 06:09:44.912082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.912011 4914 setters.go:546] "Node became not ready" node="ip-10-0-130-54.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T06:09:44Z","lastTransitionTime":"2026-11-01T06:09:44Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 01 06:09:44.937815 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.937793 4914 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-54.us-west-1.compute.internal" is owned by different node (oldNodeID="a65553c8-3856-460e-bad3-348de71cf255", newNodeID=""), cleaning up... Nov 01 06:09:44.938201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.938186 4914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:44.938247 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:44.938218 4914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.969370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.969348 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.969452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.969378 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b0505d995be0782193889599bca279d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"2b0505d995be0782193889599bca279d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.969452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.969399 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.969452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.969446 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.969566 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.969455 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b0505d995be0782193889599bca279d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"2b0505d995be0782193889599bca279d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:44.969566 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:44.969449 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c62962b404022a5a6674917fdd042e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" (UID: \"5c62962b404022a5a6674917fdd042e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:45.563027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.562986 4914 apiserver.go:52] "Watching apiserver" Nov 01 06:09:45.658551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.658525 4914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:45.658677 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.658663 4914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:45.716508 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.716490 4914 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 06:09:45.717126 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.717102 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn","openshift-network-operator/iptables-alerter-5pdw4","openshift-dns/dns-default-mkk52","openshift-network-console/networking-console-plugin-86c478bb88-82lzc","openshift-insights/insights-runtime-extractor-zf9qt","openshift-multus/multus-additional-cni-plugins-mdnbd","openshift-console-operator/console-operator-7cd56b8dbc-fqwwj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal","openshift-multus/multus-x5w6w","openshift-insights/insights-operator-7c4986d449-72qwz","openshift-image-registry/image-registry-5d4d66c7f9-2g8gv","kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal","kube-system/konnectivity-agent-l576t","openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g","kube-system/global-pull-secret-syncer-rj4mc","openshift-dns/node-resolver-v5cjs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p","openshift-ovn-kubernetes/ovnkube-node-fk9cx","openshift-image-registry/node-ca-q9jbc","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc","openshift-ingress-canary/ingress-canary-2dxsg","kube-system/kas-connection-checker-685895c7c9-lpzzd","openshift-cluster-node-tuning-operator/tuned-qmhjz","kube-system/kas-connection-checker-685895c7c9-cwffg","openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd","openshift-monitoring/cluster-monitoring-operator-884576c57-962j5","openshift-ingress/router-default-78b77c8f77-457bj","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl","openshift-multus/network-metrics-daemon-j7vht","kube-system/kas-connection-checker-685895c7c9-6tflw","openshift-network-diagnostics/network-check-target-kx8mt","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p","openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:09:45.719126 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.719110 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.719676 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.719661 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.719902 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.719881 4914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:45.720035 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.720015 4914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:45.720035 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.720032 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.720373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.720354 4914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 06:09:45.720457 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.720407 4914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-54.us-west-1.compute.internal" Nov 01 06:09:45.724294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.724256 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.724650 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.724631 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:45.729103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.729085 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:45.730068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.730052 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.731330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.731316 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.732227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.732211 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.770068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.770047 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.770068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.770047 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.772369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.772353 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 06:09:45.773976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.773963 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 01 06:09:45.774057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774041 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 01 06:09:45.774104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774070 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 01 06:09:45.774104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774095 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 01 06:09:45.774162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774117 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2658x\"" Nov 01 06:09:45.774232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774219 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 01 06:09:45.774296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774264 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 01 06:09:45.774347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774295 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 01 06:09:45.774347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774334 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jp6zc\"" Nov 01 06:09:45.774404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774348 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 01 06:09:45.774404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774349 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 01 06:09:45.774404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774384 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7z9d7\"" Nov 01 06:09:45.774404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774384 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tx6qb\"" Nov 01 06:09:45.774501 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774445 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bzrlb\"" Nov 01 06:09:45.774527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774504 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ftfjm\"" Nov 01 06:09:45.774572 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774560 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9n4lb\"" Nov 01 06:09:45.774625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774617 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-v64qp\"" Nov 01 06:09:45.774655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.774629 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rmsht\"" Nov 01 06:09:45.775855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.775842 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 06:09:45.775900 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.775884 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 06:09:45.775943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.775933 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 06:09:45.777539 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.777526 4914 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 06:09:45.820913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.820855 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 06:09:45.820913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.820876 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 06:09:45.821022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821014 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 06:09:45.821364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821347 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 01 06:09:45.821483 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821467 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rlspf\"" Nov 01 06:09:45.821540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821466 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnjvc\"" Nov 01 06:09:45.821655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821638 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdp5j\"" Nov 01 06:09:45.821727 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821694 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8pggw\"" Nov 01 06:09:45.821801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821724 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 06:09:45.821801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821754 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wr\"" Nov 01 06:09:45.821801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821769 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 06:09:45.821801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821795 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vsdrw\"" Nov 01 06:09:45.821960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821833 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8r6tp\"" Nov 01 06:09:45.821960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821874 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 06:09:45.821960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821877 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jgqp9\"" Nov 01 06:09:45.821960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821896 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m98z\"" Nov 01 06:09:45.821960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821921 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 06:09:45.821960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.821953 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kqrlq\"" Nov 01 06:09:45.822146 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822019 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-q82mm\"" Nov 01 06:09:45.822146 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822062 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78q7p\"" Nov 01 06:09:45.822146 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822076 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 06:09:45.822237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822178 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9gkn9\"" Nov 01 06:09:45.822311 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822300 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h2lht\"" Nov 01 06:09:45.822342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822313 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwtj8\"" Nov 01 06:09:45.822453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822442 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnk2x\"" Nov 01 06:09:45.822627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822615 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wss87\"" Nov 01 06:09:45.822791 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.822781 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnlbn\"" Nov 01 06:09:45.871140 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871113 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871309 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871326 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871330 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871389 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871400 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871391 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871393 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871415 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871418 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 06:09:45.871432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871434 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 06:09:45.871751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871450 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871394 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871590 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871784 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 01 06:09:45.871848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871816 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871861 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.871936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871864 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.871936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871900 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.871936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871936 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871946 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871954 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 01 06:09:45.872061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871948 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.871970 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872047 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872140 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872418 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872403 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 01 06:09:45.872491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872471 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 06:09:45.872537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872506 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872500 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872523 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872532 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872550 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872579 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872579 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872588 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872617 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872629 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872660 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 01 06:09:45.872687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872632 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 01 06:09:45.872986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872727 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 06:09:45.872986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872744 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.872986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872941 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872962 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872966 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.872986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.872964 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 01 06:09:45.873186 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.873009 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.873186 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.873017 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 06:09:45.873186 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.873047 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 06:09:45.873494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.873476 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.873633 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.873497 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.874080 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.874781 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.874821 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-var-lib-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.874855 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-etc-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.874974 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875001 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875009 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-run\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875035 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-multus-certs\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875095 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875126 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rfc\" (UniqueName: \"kubernetes.io/projected/dd901c17-d465-4a03-b320-73be6ff3d0ce-kube-api-access-p9rfc\") pod \"kas-connection-checker-685895c7c9-6tflw\" (UID: \"dd901c17-d465-4a03-b320-73be6ff3d0ce\") " pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875159 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875194 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-ovn\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875221 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-modprobe-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875365 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875376 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875421 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-socket-dir-parent\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875454 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgvz\" (UniqueName: \"kubernetes.io/projected/14525438-a3fb-4d7f-bd93-de2468852713-kube-api-access-kdgvz\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875546 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b76995-eb22-418d-9665-bb729b6e88c4-hosts-file\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875572 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55368581-d897-48e9-bd96-95d33eba4426-iptables-alerter-script\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875600 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875630 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875658 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-sys\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875683 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875712 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875738 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtbp\" (UniqueName: \"kubernetes.io/projected/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-kube-api-access-5dtbp\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875770 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8vr\" (UniqueName: \"kubernetes.io/projected/cfef5efe-9658-4dbe-a3b5-cb484fec7962-kube-api-access-8d8vr\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.875831 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876391 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.876758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876612 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876803 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876856 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876862 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nmn\" (UniqueName: \"kubernetes.io/projected/02d7b7a1-c649-45d0-9c18-14859b30ba56-kube-api-access-z8nmn\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876914 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.876989 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55368581-d897-48e9-bd96-95d33eba4426-iptables-alerter-script\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877031 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877087 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-snapshots\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877171 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgwv\" (UniqueName: \"kubernetes.io/projected/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-api-access-bmgwv\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877181 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877207 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-default-certificate\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877207 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-snapshots\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877259 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-multus\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877346 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877383 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-systemd-units\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877413 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-cnibin\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877473 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbm9\" (UniqueName: \"kubernetes.io/projected/6481f3ae-016a-4640-ac15-03e11f8963c6-kube-api-access-krbm9\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877537 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877578 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-config\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.877646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877595 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-default-certificate\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.878161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.877819 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-config\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.878672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878650 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878677 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14525438-a3fb-4d7f-bd93-de2468852713-cert\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878693 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b76995-eb22-418d-9665-bb729b6e88c4-tmp-dir\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878709 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878723 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-tmp\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878736 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxns\" (UniqueName: \"kubernetes.io/projected/80c8da36-c064-46e9-9712-b36bde260e3d-kube-api-access-sdxns\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878751 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-crio-socket\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.878769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878766 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdtv\" (UniqueName: \"kubernetes.io/projected/d7014b11-e3e2-448c-b1bc-08f298821e3b-kube-api-access-9fdtv\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878789 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878826 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b76995-eb22-418d-9665-bb729b6e88c4-tmp-dir\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878835 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c8da36-c064-46e9-9712-b36bde260e3d-tmp\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878936 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878968 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14525438-a3fb-4d7f-bd93-de2468852713-cert\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.878970 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-var-lib-kubelet\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879001 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9xp\" (UniqueName: \"kubernetes.io/projected/6c66683c-48af-4ccd-9179-f61a555c3055-kube-api-access-9g9xp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879018 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-sys-fs\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879035 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6481f3ae-016a-4640-ac15-03e11f8963c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879056 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879090 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.879117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879116 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-tmp-dir\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879137 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e862b42b-a7df-440b-9170-89c0de89c451-agent-certs\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879161 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-systemd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879177 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879191 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5zp\" (UniqueName: \"kubernetes.io/projected/6e482476-c522-476f-ad22-ab83d135404c-kube-api-access-4v5zp\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879205 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-kubelet\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879205 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-tmp-dir\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879218 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-conf-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879233 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mh9f\" (UniqueName: \"kubernetes.io/projected/b2b76995-eb22-418d-9665-bb729b6e88c4-kube-api-access-2mh9f\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879313 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-host\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879318 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879334 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-socket-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879356 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e862b42b-a7df-440b-9170-89c0de89c451-konnectivity-ca\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879370 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e862b42b-a7df-440b-9170-89c0de89c451-agent-certs\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879423 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879443 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-slash\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879468 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-log-socket\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879486 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879501 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879486 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e862b42b-a7df-440b-9170-89c0de89c451-konnectivity-ca\") pod \"konnectivity-agent-l576t\" (UID: \"e862b42b-a7df-440b-9170-89c0de89c451\") " pod="kube-system/konnectivity-agent-l576t" Nov 01 06:09:45.879621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879541 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879643 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-kubernetes\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879671 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-service-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879718 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-bin\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879740 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-env-overrides\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879785 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-service-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879798 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2x6\" (UniqueName: \"kubernetes.io/projected/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-kube-api-access-hl2x6\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879822 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-system-cni-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879876 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-serviceca\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879908 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f7226f4-6e78-4172-8f28-c3225d93636f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879925 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-env-overrides\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879962 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68vq\" (UniqueName: \"kubernetes.io/projected/cc06a215-884f-40b1-90fc-208bcc319a76-kube-api-access-l68vq\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.879990 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-conf\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880020 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssnx\" (UniqueName: \"kubernetes.io/projected/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-kube-api-access-kssnx\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880025 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-serviceca\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880053 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-system-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880085 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f7226f4-6e78-4172-8f28-c3225d93636f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880134 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cni-binary-copy\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880160 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-k8s-cni-cncf-io\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880191 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-dbus\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880209 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55368581-d897-48e9-bd96-95d33eba4426-host-slash\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880231 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47jb\" (UniqueName: \"kubernetes.io/projected/78c096fb-88d1-4f4c-813d-b2095302bf8c-kube-api-access-x47jb\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880270 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cni-binary-copy\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880309 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.880337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880341 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880365 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880390 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-kubelet\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880425 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880463 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880482 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880507 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880523 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-bin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880537 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880550 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-netd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880565 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-script-lib\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880583 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-lib-modules\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880623 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-tmp\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880656 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880674 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-tmp\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880685 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c66683c-48af-4ccd-9179-f61a555c3055-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880711 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880736 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78c096fb-88d1-4f4c-813d-b2095302bf8c-serving-cert\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880784 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrsr\" (UniqueName: \"kubernetes.io/projected/99cae588-1b8c-4972-bda1-781a3edb687a-kube-api-access-hnrsr\") pod \"kas-connection-checker-685895c7c9-lpzzd\" (UID: \"99cae588-1b8c-4972-bda1-781a3edb687a\") " pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880792 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-script-lib\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880870 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthh4\" (UniqueName: \"kubernetes.io/projected/8fa19977-5c71-40c4-acb8-29e0d569efb4-kube-api-access-qthh4\") pod \"volume-data-source-validator-58c9f5f46d-4fn7p\" (UID: \"8fa19977-5c71-40c4-acb8-29e0d569efb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880900 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880936 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c66683c-48af-4ccd-9179-f61a555c3055-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880953 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78c096fb-88d1-4f4c-813d-b2095302bf8c-serving-cert\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.880961 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c66683c-48af-4ccd-9179-f61a555c3055-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881025 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.881096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881068 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881141 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c66683c-48af-4ccd-9179-f61a555c3055-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881223 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-hostroot\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881254 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-etc-selinux\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881292 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovn-node-metrics-cert\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881317 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nwm\" (UniqueName: \"kubernetes.io/projected/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-kube-api-access-x8nwm\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881342 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-systemd\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881364 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881390 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-daemon-config\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881408 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-etc-kubernetes\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881426 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gkc\" (UniqueName: \"kubernetes.io/projected/8079984d-980a-4287-826e-91194cf07e9e-kube-api-access-l5gkc\") pod \"kas-connection-checker-685895c7c9-cwffg\" (UID: \"8079984d-980a-4287-826e-91194cf07e9e\") " pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881447 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovn-node-metrics-cert\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881453 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881514 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-data-volume\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881533 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-daemon-config\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881587 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-stats-auth\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881677 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-tuned\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881686 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-data-volume\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881716 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881720 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-stats-auth\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881732 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-tuned\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881756 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cnibin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881771 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-kubelet-config\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881793 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snstj\" (UniqueName: \"kubernetes.io/projected/55368581-d897-48e9-bd96-95d33eba4426-kube-api-access-snstj\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881807 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-config\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881823 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxck\" (UniqueName: \"kubernetes.io/projected/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-kube-api-access-wfxck\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881897 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-netns\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881934 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881959 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjgw\" (UniqueName: \"kubernetes.io/projected/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a-kube-api-access-fmjgw\") pod \"migrator-66df5d496b-qq59g\" (UID: \"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881960 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-ovnkube-config\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.881994 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.882068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882013 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fms\" (UniqueName: \"kubernetes.io/projected/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-kube-api-access-m9fms\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882123 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42fs\" (UniqueName: \"kubernetes.io/projected/0e300938-7fc7-451b-9ca5-899929edea57-kube-api-access-b42fs\") pod \"network-check-source-6f78d59f99-bskfd\" (UID: \"0e300938-7fc7-451b-9ca5-899929edea57\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882210 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882227 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-netns\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882259 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-node-log\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882327 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-registration-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882329 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882384 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cfef5efe-9658-4dbe-a3b5-cb484fec7962-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882414 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-host\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882444 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysconfig\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882482 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-config-volume\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882500 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c8da36-c064-46e9-9712-b36bde260e3d-serving-cert\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882514 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-device-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882531 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78c096fb-88d1-4f4c-813d-b2095302bf8c-config\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882560 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-serving-cert\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882582 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882605 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cfef5efe-9658-4dbe-a3b5-cb484fec7962-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882677 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/80c8da36-c064-46e9-9712-b36bde260e3d-serving-cert\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882689 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-config-volume\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882784 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78c096fb-88d1-4f4c-813d-b2095302bf8c-config\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882808 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-serving-cert\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.883115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.882865 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-original-pull-secret\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.922682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.922659 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 01 06:09:45.922743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.922720 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 06:09:45.923326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.923312 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 06:09:45.927853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.927836 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.930065 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.930050 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.930139 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.930081 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-trusted-ca\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.931850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.931828 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c8da36-c064-46e9-9712-b36bde260e3d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.932322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932298 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgwv\" (UniqueName: \"kubernetes.io/projected/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-api-access-bmgwv\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.932322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932318 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8vr\" (UniqueName: \"kubernetes.io/projected/cfef5efe-9658-4dbe-a3b5-cb484fec7962-kube-api-access-8d8vr\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.932415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932338 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rfc\" (UniqueName: \"kubernetes.io/projected/dd901c17-d465-4a03-b320-73be6ff3d0ce-kube-api-access-p9rfc\") pod \"kas-connection-checker-685895c7c9-6tflw\" (UID: \"dd901c17-d465-4a03-b320-73be6ff3d0ce\") " pod="kube-system/kas-connection-checker-685895c7c9-6tflw" Nov 01 06:09:45.932478 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932459 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgvz\" (UniqueName: \"kubernetes.io/projected/14525438-a3fb-4d7f-bd93-de2468852713-kube-api-access-kdgvz\") pod \"ingress-canary-2dxsg\" (UID: \"14525438-a3fb-4d7f-bd93-de2468852713\") " pod="openshift-ingress-canary/ingress-canary-2dxsg" Nov 01 06:09:45.932634 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932620 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.932762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932743 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nmn\" (UniqueName: \"kubernetes.io/projected/02d7b7a1-c649-45d0-9c18-14859b30ba56-kube-api-access-z8nmn\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.932791 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.932746 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtbp\" (UniqueName: \"kubernetes.io/projected/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-kube-api-access-5dtbp\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.933799 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.933781 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbm9\" (UniqueName: \"kubernetes.io/projected/6481f3ae-016a-4640-ac15-03e11f8963c6-kube-api-access-krbm9\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.934898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.934878 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9xp\" (UniqueName: \"kubernetes.io/projected/6c66683c-48af-4ccd-9179-f61a555c3055-kube-api-access-9g9xp\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-js7kl\" (UID: \"6c66683c-48af-4ccd-9179-f61a555c3055\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" Nov 01 06:09:45.935010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.934993 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdtv\" (UniqueName: \"kubernetes.io/projected/d7014b11-e3e2-448c-b1bc-08f298821e3b-kube-api-access-9fdtv\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:45.935149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.935134 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxns\" (UniqueName: \"kubernetes.io/projected/80c8da36-c064-46e9-9712-b36bde260e3d-kube-api-access-sdxns\") pod \"insights-operator-7c4986d449-72qwz\" (UID: \"80c8da36-c064-46e9-9712-b36bde260e3d\") " pod="openshift-insights/insights-operator-7c4986d449-72qwz" Nov 01 06:09:45.939334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.939313 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.939803 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.939782 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mh9f\" (UniqueName: \"kubernetes.io/projected/b2b76995-eb22-418d-9665-bb729b6e88c4-kube-api-access-2mh9f\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.939874 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.939859 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fms\" (UniqueName: \"kubernetes.io/projected/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-kube-api-access-m9fms\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.939919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.939904 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qj2h\" (UniqueName: \"kubernetes.io/projected/62a9bf0c-6122-4683-80bf-43370ac319a4-kube-api-access-9qj2h\") pod \"network-check-target-kx8mt\" (UID: \"62a9bf0c-6122-4683-80bf-43370ac319a4\") " pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:45.939984 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.939970 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5zp\" (UniqueName: \"kubernetes.io/projected/6e482476-c522-476f-ad22-ab83d135404c-kube-api-access-4v5zp\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.941617 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.941594 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxck\" (UniqueName: \"kubernetes.io/projected/1b4526c0-30f7-4558-b9e6-4a949e19e8fb-kube-api-access-wfxck\") pod \"console-operator-7cd56b8dbc-fqwwj\" (UID: \"1b4526c0-30f7-4558-b9e6-4a949e19e8fb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:45.941847 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.941831 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47jb\" (UniqueName: \"kubernetes.io/projected/78c096fb-88d1-4f4c-813d-b2095302bf8c-kube-api-access-x47jb\") pod \"service-ca-operator-544d749c8c-jqmxn\" (UID: \"78c096fb-88d1-4f4c-813d-b2095302bf8c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" Nov 01 06:09:45.941888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.941834 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gkc\" (UniqueName: \"kubernetes.io/projected/8079984d-980a-4287-826e-91194cf07e9e-kube-api-access-l5gkc\") pod \"kas-connection-checker-685895c7c9-cwffg\" (UID: \"8079984d-980a-4287-826e-91194cf07e9e\") " pod="kube-system/kas-connection-checker-685895c7c9-cwffg" Nov 01 06:09:45.941888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.941838 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2x6\" (UniqueName: \"kubernetes.io/projected/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-kube-api-access-hl2x6\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.942529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942381 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssnx\" (UniqueName: \"kubernetes.io/projected/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-kube-api-access-kssnx\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.942529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942470 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nwm\" (UniqueName: \"kubernetes.io/projected/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-kube-api-access-x8nwm\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.942529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942479 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjgw\" (UniqueName: \"kubernetes.io/projected/b8a138fd-e92b-4ac3-97ad-c6ad969cc43a-kube-api-access-fmjgw\") pod \"migrator-66df5d496b-qq59g\" (UID: \"b8a138fd-e92b-4ac3-97ad-c6ad969cc43a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qq59g" Nov 01 06:09:45.942529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942504 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snstj\" (UniqueName: \"kubernetes.io/projected/55368581-d897-48e9-bd96-95d33eba4426-kube-api-access-snstj\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.942674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942595 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.942674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942600 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42fs\" (UniqueName: \"kubernetes.io/projected/0e300938-7fc7-451b-9ca5-899929edea57-kube-api-access-b42fs\") pod \"network-check-source-6f78d59f99-bskfd\" (UID: \"0e300938-7fc7-451b-9ca5-899929edea57\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bskfd" Nov 01 06:09:45.942786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942771 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrsr\" (UniqueName: \"kubernetes.io/projected/99cae588-1b8c-4972-bda1-781a3edb687a-kube-api-access-hnrsr\") pod \"kas-connection-checker-685895c7c9-lpzzd\" (UID: \"99cae588-1b8c-4972-bda1-781a3edb687a\") " pod="kube-system/kas-connection-checker-685895c7c9-lpzzd" Nov 01 06:09:45.942815 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942789 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68vq\" (UniqueName: \"kubernetes.io/projected/cc06a215-884f-40b1-90fc-208bcc319a76-kube-api-access-l68vq\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.942842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942793 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthh4\" (UniqueName: \"kubernetes.io/projected/8fa19977-5c71-40c4-acb8-29e0d569efb4-kube-api-access-qthh4\") pod \"volume-data-source-validator-58c9f5f46d-4fn7p\" (UID: \"8fa19977-5c71-40c4-acb8-29e0d569efb4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4fn7p" Nov 01 06:09:45.942953 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.942939 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.982914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982895 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.982990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982918 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-kubelet\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.982990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982934 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.982990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982948 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-bin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.982990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982966 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.982990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982987 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-netd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.982988 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-kubelet\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983026 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-bin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983026 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-netd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983040 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-lib-modules\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983044 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983066 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983068 4914 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983088 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-hostroot\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983107 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-etc-selinux\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983130 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-systemd\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983140 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-hostroot\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983083 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983154 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983144 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-lib-modules\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983166 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-etc-selinux\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983176 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-systemd\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983188 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983133 4914 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983201 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983179 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.483158535 +0000 UTC m=+2.035947730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983256 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.48324521 +0000 UTC m=+2.036034381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983267 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-etc-kubernetes\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983300 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983314 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cnibin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983326 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-kubelet-config\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983339 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-netns\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983352 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983366 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-netns\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983381 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-node-log\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983401 4914 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983415 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-kubelet-config\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983418 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-etc-kubernetes\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983439 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-run-netns\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983445 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-registration-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983402 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-registration-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983455 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-cnibin\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983469 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.483456111 +0000 UTC m=+2.036245282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : secret "metrics-daemon-secret" not found Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983468 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-node-log\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983483 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983502 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-host\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983512 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-netns\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983531 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-host\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983525 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysconfig\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983566 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-device-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983575 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysconfig\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983594 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983617 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-device-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983621 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-var-lib-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983633 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983644 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-etc-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983662 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-run\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983670 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-var-lib-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983676 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-multus-certs\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983693 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-multus-certs\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983703 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983707 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-etc-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.983700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983718 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-ovn\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983720 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-run\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983750 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-openvswitch\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983751 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-ovn\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983781 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-modprobe-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983796 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983810 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-socket-dir-parent\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983823 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b76995-eb22-418d-9665-bb729b6e88c4-hosts-file\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983836 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983848 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-sys\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983864 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983882 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2b76995-eb22-418d-9665-bb729b6e88c4-hosts-file\") pod \"node-resolver-v5cjs\" (UID: \"b2b76995-eb22-418d-9665-bb729b6e88c4\") " pod="openshift-dns/node-resolver-v5cjs" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983882 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-socket-dir-parent\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983892 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-multus\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983901 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-sys\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983912 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983924 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983927 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-cni-multus\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983932 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-systemd-units\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983938 4914 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983964 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.483952392 +0000 UTC m=+2.036741598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983973 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-systemd-units\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983976 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-cnibin\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.983944 4914 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983847 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-modprobe-d\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.983999 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-cnibin\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984007 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.483993568 +0000 UTC m=+2.036782740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984022 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984031 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984038 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.48401866 +0000 UTC m=+2.036807831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984066 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984080 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984097 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-crio-socket\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984111 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984124 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-var-lib-kubelet\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984127 4914 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984138 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-sys-fs\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984149 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.4841428 +0000 UTC m=+2.036932028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984164 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-sys-fs\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984169 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.484154828 +0000 UTC m=+2.036944035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984186 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984192 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984195 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-var-lib-kubelet\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984201 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-systemd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984209 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.484203835 +0000 UTC m=+2.036993058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984225 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-kubelet\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984236 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-run-systemd\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984239 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-conf-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984208 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-crio-socket\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984268 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-multus-conf-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984266 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-var-lib-kubelet\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984290 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-host\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984315 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-socket-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984338 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-slash\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984359 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-log-socket\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984369 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1-host\") pod \"node-ca-q9jbc\" (UID: \"cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1\") " pod="openshift-image-registry/node-ca-q9jbc" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984383 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984378 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-slash\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984393 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02d7b7a1-c649-45d0-9c18-14859b30ba56-socket-dir\") pod \"aws-ebs-csi-driver-node-l4x2p\" (UID: \"02d7b7a1-c649-45d0-9c18-14859b30ba56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l4x2p" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984407 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-log-socket\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984411 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-kubernetes\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984436 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-bin\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984444 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-kubernetes\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984444 4914 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984460 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-system-cni-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984473 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.484466353 +0000 UTC m=+2.037255577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : configmap "kube-rbac-proxy" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984480 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1caa2f5d-b542-4af5-8dc4-3b75ab99c08f-host-cni-bin\") pod \"ovnkube-node-fk9cx\" (UID: \"1caa2f5d-b542-4af5-8dc4-3b75ab99c08f\") " pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984494 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-conf\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984500 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6481f3ae-016a-4640-ac15-03e11f8963c6-system-cni-dir\") pod \"multus-additional-cni-plugins-mdnbd\" (UID: \"6481f3ae-016a-4640-ac15-03e11f8963c6\") " pod="openshift-multus/multus-additional-cni-plugins-mdnbd" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984509 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-system-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984530 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-k8s-cni-cncf-io\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984538 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-system-cni-dir\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984546 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-dbus\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984561 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55368581-d897-48e9-bd96-95d33eba4426-host-slash\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984572 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0846b5f6-3f41-45a7-b3c4-0402f250d3c8-host-run-k8s-cni-cncf-io\") pod \"multus-x5w6w\" (UID: \"0846b5f6-3f41-45a7-b3c4-0402f250d3c8\") " pod="openshift-multus/multus-x5w6w" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984575 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984612 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc-etc-sysctl-conf\") pod \"tuned-qmhjz\" (UID: \"84d1ba6a-d76e-4f87-84ec-3ddebb04b0cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qmhjz" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984619 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55368581-d897-48e9-bd96-95d33eba4426-host-slash\") pod \"iptables-alerter-5pdw4\" (UID: \"55368581-d897-48e9-bd96-95d33eba4426\") " pod="openshift-network-operator/iptables-alerter-5pdw4" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:45.984643 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a-dbus\") pod \"global-pull-secret-syncer-rj4mc\" (UID: \"1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a\") " pod="kube-system/global-pull-secret-syncer-rj4mc" Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984680 4914 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:45.984971 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:45.984719 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:46.484706356 +0000 UTC m=+2.037495530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:46.019225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.019205 4914 scope.go:122] "RemoveContainer" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" Nov 01 06:09:46.489215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489157 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:46.489215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489193 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489230 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489264 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489313 4914 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489331 4914 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489333 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489378 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.48936417 +0000 UTC m=+3.042153362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489380 4914 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:46.489401 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489403 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489393528 +0000 UTC m=+3.042182713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489405 4914 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489431 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489440 4914 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489449 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489462 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489447482 +0000 UTC m=+3.042236682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489473 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489482 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489482 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489473074 +0000 UTC m=+3.042262251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489495 4914 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489503 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489495042 +0000 UTC m=+3.042284220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : secret "metrics-daemon-secret" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489521 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489513311 +0000 UTC m=+3.042302490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489536 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489528354 +0000 UTC m=+3.042317538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489579 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489607 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489636 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.489674 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489686 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489674817 +0000 UTC m=+3.042464026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489699 4914 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489715 4914 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489756 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489746005 +0000 UTC m=+3.042535199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : configmap "kube-rbac-proxy" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489770 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489779 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:46.489788 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489771 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489763024 +0000 UTC m=+3.042552203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:46.490459 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.489810 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:47.489800989 +0000 UTC m=+3.042590160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:46.663655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.663634 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:09:46.663996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.663983 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/1.log" Nov 01 06:09:46.664042 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.664014 4914 generic.go:356] "Generic (PLEG): container finished" podID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" containerID="77e5e6f26858bc7970ad2345f7369b0cc76cda490eeff1416b1c9928d910dfef" exitCode=255 Nov 01 06:09:46.664084 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.664067 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" event={"ID":"1b4526c0-30f7-4558-b9e6-4a949e19e8fb","Type":"ContainerDied","Data":"77e5e6f26858bc7970ad2345f7369b0cc76cda490eeff1416b1c9928d910dfef"} Nov 01 06:09:46.664129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.664120 4914 scope.go:122] "RemoveContainer" containerID="df44f944cdeec103eebf679d72b3b39c8af56f2d542dee485b9db0e0a7badaca" Nov 01 06:09:46.664547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:46.664527 4914 scope.go:122] "RemoveContainer" containerID="77e5e6f26858bc7970ad2345f7369b0cc76cda490eeff1416b1c9928d910dfef" Nov 01 06:09:46.664686 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:46.664667 4914 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fqwwj_openshift-console-operator(1b4526c0-30f7-4558-b9e6-4a949e19e8fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" Nov 01 06:09:47.225883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.225862 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5cjs_b2b76995-eb22-418d-9665-bb729b6e88c4/dns-node-resolver/0.log" Nov 01 06:09:47.498122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498080 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:47.498122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498108 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:47.498253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498125 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:47.498253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498145 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:47.498253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498161 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:47.498253 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498225 4914 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:47.498253 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498230 4914 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498260 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498298 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498265694 +0000 UTC m=+5.051054868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498239 4914 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498309 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498318 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498337 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498323679 +0000 UTC m=+5.051112851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498340 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498353 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498355 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498346921 +0000 UTC m=+5.051136096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498370 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls podName:6e482476-c522-476f-ad22-ab83d135404c nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.49836389 +0000 UTC m=+5.051153060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-g4hrc" (UID: "6e482476-c522-476f-ad22-ab83d135404c") : secret "samples-operator-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498380 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498376132 +0000 UTC m=+5.051165302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498391 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498386627 +0000 UTC m=+5.051175798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498416 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498440 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:47.498455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498456 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498477 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498498 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498547 4914 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498547 4914 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498569 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498562328 +0000 UTC m=+5.051351523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498570 4914 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498576 4914 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498593 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498582231 +0000 UTC m=+5.051371401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498609 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls podName:cfef5efe-9658-4dbe-a3b5-cb484fec7962 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498598646 +0000 UTC m=+5.051387841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-962j5" (UID: "cfef5efe-9658-4dbe-a3b5-cb484fec7962") : secret "cluster-monitoring-operator-tls" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:47.498621 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs podName:d7014b11-e3e2-448c-b1bc-08f298821e3b nodeName:}" failed. No retries permitted until 2026-11-01 06:09:49.498615349 +0000 UTC m=+5.051404520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs") pod "network-metrics-daemon-j7vht" (UID: "d7014b11-e3e2-448c-b1bc-08f298821e3b") : secret "metrics-daemon-secret" not found Nov 01 06:09:47.499116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.498982 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:47.670269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.670246 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:09:47.825560 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:47.825511 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9jbc_cb8ebdf3-bc06-4ad6-b328-a4b1f973e9d1/node-ca/0.log" Nov 01 06:09:48.470439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:48.470419 4914 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:48.470713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:48.470701 4914 scope.go:122] "RemoveContainer" containerID="77e5e6f26858bc7970ad2345f7369b0cc76cda490eeff1416b1c9928d910dfef" Nov 01 06:09:48.470841 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:48.470826 4914 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fqwwj_openshift-console-operator(1b4526c0-30f7-4558-b9e6-4a949e19e8fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" Nov 01 06:09:48.626732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:48.626712 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2dxsg_14525438-a3fb-4d7f-bd93-de2468852713/serve-healthcheck-canary/0.log" Nov 01 06:09:49.327119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.327098 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:49.327408 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.327371 4914 scope.go:122] "RemoveContainer" containerID="77e5e6f26858bc7970ad2345f7369b0cc76cda490eeff1416b1c9928d910dfef" Nov 01 06:09:49.327500 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.327485 4914 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-fqwwj_openshift-console-operator(1b4526c0-30f7-4558-b9e6-4a949e19e8fb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" Nov 01 06:09:49.426532 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.426515 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-qq59g_b8a138fd-e92b-4ac3-97ad-c6ad969cc43a/migrator/0.log" Nov 01 06:09:49.515254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515231 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:49.515353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515259 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:49.515353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515297 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:49.515353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515330 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:49.515444 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515359 4914 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:49.515444 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515399 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.51538773 +0000 UTC m=+9.068176901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:49.515444 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515403 4914 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:49.515557 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515447 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.515433133 +0000 UTC m=+9.068222307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:49.515557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515469 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:49.515557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515496 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:49.515557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515521 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515557 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515594 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515607 4914 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.515621 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515638 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.515625061 +0000 UTC m=+9.068414244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : configmap references non-existent config key: service-ca.crt Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515649 4914 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515690 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515694 4914 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515701 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf65c6b-wb9t6: secret "image-registry-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515707 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.515690987 +0000 UTC m=+9.068480175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515709 4914 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d4d66c7f9-2g8gv: secret "image-registry-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515724 4914 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515735 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls podName:ac8f514b-f92b-4e2c-b097-6594f9c2c7ea nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.515724857 +0000 UTC m=+9.068514031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls") pod "image-registry-698cf65c6b-wb9t6" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea") : secret "image-registry-tls" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515774 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.51576095 +0000 UTC m=+9.068550150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:49.515794 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:49.515790 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls podName:ef1e9b60-c1a2-4e84-b155-d43c5b023617 nodeName:}" failed. No retries permitted until 2026-11-01 06:09:53.515780743 +0000 UTC m=+9.068569913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls") pod "image-registry-5d4d66c7f9-2g8gv" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617") : secret "image-registry-tls" not found Nov 01 06:09:49.517868 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.517848 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cfef5efe-9658-4dbe-a3b5-cb484fec7962-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-962j5\" (UID: \"cfef5efe-9658-4dbe-a3b5-cb484fec7962\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:49.517967 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.517929 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7014b11-e3e2-448c-b1bc-08f298821e3b-metrics-certs\") pod \"network-metrics-daemon-j7vht\" (UID: \"d7014b11-e3e2-448c-b1bc-08f298821e3b\") " pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:49.518083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.518061 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e482476-c522-476f-ad22-ab83d135404c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-g4hrc\" (UID: \"6e482476-c522-476f-ad22-ab83d135404c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:49.620684 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.620663 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" Nov 01 06:09:49.620765 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.620669 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" Nov 01 06:09:49.627071 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.627050 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-js7kl_6c66683c-48af-4ccd-9179-f61a555c3055/kube-storage-version-migrator-operator/0.log" Nov 01 06:09:49.629308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:49.629295 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7vht" Nov 01 06:09:50.068565 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.068511 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc"] Nov 01 06:09:50.070960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.070938 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-962j5"] Nov 01 06:09:50.071537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.071517 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j7vht"] Nov 01 06:09:50.110770 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:50.110747 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfef5efe_9658_4dbe_a3b5_cb484fec7962.slice/crio-57a2a6b3337593f4fba398eedbc5d5afe158f8ecb8c900a8b5fcf4dfa29cd60a WatchSource:0}: Error finding container 57a2a6b3337593f4fba398eedbc5d5afe158f8ecb8c900a8b5fcf4dfa29cd60a: Status 404 returned error can't find the container with id 57a2a6b3337593f4fba398eedbc5d5afe158f8ecb8c900a8b5fcf4dfa29cd60a Nov 01 06:09:50.111458 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:50.111443 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7014b11_e3e2_448c_b1bc_08f298821e3b.slice/crio-9ff40d487c3608da7264248721062fad1e8df0f70d2c50d7afe14851ec64c57a WatchSource:0}: Error finding container 9ff40d487c3608da7264248721062fad1e8df0f70d2c50d7afe14851ec64c57a: Status 404 returned error can't find the container with id 9ff40d487c3608da7264248721062fad1e8df0f70d2c50d7afe14851ec64c57a Nov 01 06:09:50.113303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.113255 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:09:50.355570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.355544 4914 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l576t_e862b42b-a7df-440b-9170-89c0de89c451/konnectivity-agent/0.log" limit=1024 Nov 01 06:09:50.680519 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.680470 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" event={"ID":"cfef5efe-9658-4dbe-a3b5-cb484fec7962","Type":"ContainerStarted","Data":"57a2a6b3337593f4fba398eedbc5d5afe158f8ecb8c900a8b5fcf4dfa29cd60a"} Nov 01 06:09:50.681553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.681523 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j7vht" event={"ID":"d7014b11-e3e2-448c-b1bc-08f298821e3b","Type":"ContainerStarted","Data":"9ff40d487c3608da7264248721062fad1e8df0f70d2c50d7afe14851ec64c57a"} Nov 01 06:09:50.682525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:50.682508 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" event={"ID":"6e482476-c522-476f-ad22-ab83d135404c","Type":"ContainerStarted","Data":"87301691d11dc8d1b2bdf61215f51d1d4c4a9dbb2097facdb848d107e356ae1c"} Nov 01 06:09:51.001313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:51.001254 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:09:51.881892 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:51.881866 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:52.045686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.045616 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:52.696069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.695995 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j7vht" event={"ID":"d7014b11-e3e2-448c-b1bc-08f298821e3b","Type":"ContainerStarted","Data":"6a042c76fd905f1b5c5c85c0c4732e1adbce1701d48f5cac122093d416170ab4"} Nov 01 06:09:52.696199 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.696078 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j7vht" event={"ID":"d7014b11-e3e2-448c-b1bc-08f298821e3b","Type":"ContainerStarted","Data":"27697199acad629cac5cab3fbb3b994938f1fdac6029810306f0b088792ca86c"} Nov 01 06:09:52.697773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.697745 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" event={"ID":"6e482476-c522-476f-ad22-ab83d135404c","Type":"ContainerStarted","Data":"8584abee843c727ae0bc961380362d02f7ab1479bc1ed55d6783d4c440805088"} Nov 01 06:09:52.697893 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.697787 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-g4hrc" event={"ID":"6e482476-c522-476f-ad22-ab83d135404c","Type":"ContainerStarted","Data":"076a010fa262ee75a49215c0fee3ffd2a446d7fb75435038190b9063df30d2c5"} Nov 01 06:09:52.699051 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.699025 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-962j5" event={"ID":"cfef5efe-9658-4dbe-a3b5-cb484fec7962","Type":"ContainerStarted","Data":"5eaaae049121830c00221a0d175dd8bdf1973fcacb12ddca0cf01f6f6c708385"} Nov 01 06:09:52.699154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:52.699119 4914 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 06:09:53.548153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548123 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548158 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548182 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548205 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548234 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548255 4914 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548303 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548329 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548343 4914 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548389 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert podName:8f7226f4-6e78-4172-8f28-c3225d93636f nodeName:}" failed. No retries permitted until 2026-11-01 06:10:01.548369834 +0000 UTC m=+17.101159020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-82lzc" (UID: "8f7226f4-6e78-4172-8f28-c3225d93636f") : secret "networking-console-plugin-cert" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548406 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls podName:4fe4277a-6d12-4b3c-aaef-1abd09159ac6 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:01.548397218 +0000 UTC m=+17.101186389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls") pod "dns-default-mkk52" (UID: "4fe4277a-6d12-4b3c-aaef-1abd09159ac6") : secret "dns-default-metrics-tls" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548447 4914 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548464 4914 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548494 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs podName:cc06a215-884f-40b1-90fc-208bcc319a76 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:01.548484852 +0000 UTC m=+17.101274057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs") pod "router-default-78b77c8f77-457bj" (UID: "cc06a215-884f-40b1-90fc-208bcc319a76") : secret "router-metrics-certs-default" not found Nov 01 06:09:53.548511 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:09:53.548515 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls podName:7fe371b4-16fb-42ce-bdd7-5e8116bcc506 nodeName:}" failed. No retries permitted until 2026-11-01 06:10:01.548499705 +0000 UTC m=+17.101288892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zf9qt" (UID: "7fe371b4-16fb-42ce-bdd7-5e8116bcc506") : secret "insights-runtime-extractor-tls" not found Nov 01 06:09:53.549051 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.548858 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc06a215-884f-40b1-90fc-208bcc319a76-service-ca-bundle\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:09:53.550618 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.550594 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"image-registry-698cf65c6b-wb9t6\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:53.550711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.550601 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"image-registry-5d4d66c7f9-2g8gv\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:53.776999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.776978 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fk9cx" Nov 01 06:09:53.825317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.825255 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:53.831014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:53.830997 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:54.243092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.243069 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d4d66c7f9-2g8gv"] Nov 01 06:09:54.245773 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:54.245748 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1e9b60_c1a2_4e84_b155_d43c5b023617.slice/crio-f0acbf064593c229c5aa9db745d27a85c9cb2f26e2cb0472031b5caa99949b15 WatchSource:0}: Error finding container f0acbf064593c229c5aa9db745d27a85c9cb2f26e2cb0472031b5caa99949b15: Status 404 returned error can't find the container with id f0acbf064593c229c5aa9db745d27a85c9cb2f26e2cb0472031b5caa99949b15 Nov 01 06:09:54.250153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.250128 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:09:54.253975 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:09:54.253952 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f514b_f92b_4e2c_b097_6594f9c2c7ea.slice/crio-d3322e6e98647f909d3782c4402b402480558aaa45f430bc2683d7c572852261 WatchSource:0}: Error finding container d3322e6e98647f909d3782c4402b402480558aaa45f430bc2683d7c572852261: Status 404 returned error can't find the container with id d3322e6e98647f909d3782c4402b402480558aaa45f430bc2683d7c572852261 Nov 01 06:09:54.707324 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.706738 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" event={"ID":"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea","Type":"ContainerStarted","Data":"6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43"} Nov 01 06:09:54.707324 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.706818 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" event={"ID":"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea","Type":"ContainerStarted","Data":"d3322e6e98647f909d3782c4402b402480558aaa45f430bc2683d7c572852261"} Nov 01 06:09:54.708239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.708216 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" event={"ID":"ef1e9b60-c1a2-4e84-b155-d43c5b023617","Type":"ContainerStarted","Data":"0cde7846097fc9bfc860d3ba62d7657edc0d6f06756a1fa8b8031cc5012a707c"} Nov 01 06:09:54.708347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.708257 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" event={"ID":"ef1e9b60-c1a2-4e84-b155-d43c5b023617","Type":"ContainerStarted","Data":"f0acbf064593c229c5aa9db745d27a85c9cb2f26e2cb0472031b5caa99949b15"} Nov 01 06:09:54.708347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:54.708315 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:09:55.196246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:55.196227 4914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-54.us-west-1.compute.internal" event="NodeReady" Nov 01 06:09:55.711338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:55.711316 4914 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 06:09:58.284656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:58.284632 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:09:59.609395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:59.609367 4914 scope.go:122] "RemoveContainer" containerID="77e5e6f26858bc7970ad2345f7369b0cc76cda490eeff1416b1c9928d910dfef" Nov 01 06:09:59.726388 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:59.726369 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:09:59.726499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:59.726411 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" event={"ID":"1b4526c0-30f7-4558-b9e6-4a949e19e8fb","Type":"ContainerStarted","Data":"2c084d05665a8645ecb42542e6bd717b5b7db3ac4ace203b8cb95eaab191820a"} Nov 01 06:09:59.726611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:59.726593 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:09:59.727459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:59.727436 4914 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-fqwwj container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body= Nov 01 06:09:59.727551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:09:59.727475 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" podUID="1b4526c0-30f7-4558-b9e6-4a949e19e8fb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" Nov 01 06:10:00.311540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:00.311516 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:10:00.733880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:00.733852 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-fqwwj" Nov 01 06:10:01.618169 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.618138 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:10:01.618169 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.618170 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:01.618426 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.618228 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:10:01.618426 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.618261 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:10:01.620714 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.620683 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fe4277a-6d12-4b3c-aaef-1abd09159ac6-metrics-tls\") pod \"dns-default-mkk52\" (UID: \"4fe4277a-6d12-4b3c-aaef-1abd09159ac6\") " pod="openshift-dns/dns-default-mkk52" Nov 01 06:10:01.620875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.620721 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fe371b4-16fb-42ce-bdd7-5e8116bcc506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zf9qt\" (UID: \"7fe371b4-16fb-42ce-bdd7-5e8116bcc506\") " pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:10:01.620943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.620883 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f7226f4-6e78-4172-8f28-c3225d93636f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-82lzc\" (UID: \"8f7226f4-6e78-4172-8f28-c3225d93636f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:10:01.620943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.620887 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc06a215-884f-40b1-90fc-208bcc319a76-metrics-certs\") pod \"router-default-78b77c8f77-457bj\" (UID: \"cc06a215-884f-40b1-90fc-208bcc319a76\") " pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:01.625668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.625652 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" Nov 01 06:10:01.632204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.632187 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkk52" Nov 01 06:10:01.633311 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.633290 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zf9qt" Nov 01 06:10:01.920420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:01.920363 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:02.000219 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.000194 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-82lzc"] Nov 01 06:10:02.042731 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:10:02.042708 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7226f4_6e78_4172_8f28_c3225d93636f.slice/crio-ef103cd6b15b6fefde99f2cda62d0a64ade293c907ed59218ad5a283fda460cd WatchSource:0}: Error finding container ef103cd6b15b6fefde99f2cda62d0a64ade293c907ed59218ad5a283fda460cd: Status 404 returned error can't find the container with id ef103cd6b15b6fefde99f2cda62d0a64ade293c907ed59218ad5a283fda460cd Nov 01 06:10:02.051548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.051525 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zf9qt"] Nov 01 06:10:02.053736 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:10:02.053711 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe371b4_16fb_42ce_bdd7_5e8116bcc506.slice/crio-b4022c3df45da84d3ba73f601a6f9af12e0994985ec3b11623d422721581ce22 WatchSource:0}: Error finding container b4022c3df45da84d3ba73f601a6f9af12e0994985ec3b11623d422721581ce22: Status 404 returned error can't find the container with id b4022c3df45da84d3ba73f601a6f9af12e0994985ec3b11623d422721581ce22 Nov 01 06:10:02.092879 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:10:02.092854 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe4277a_6d12_4b3c_aaef_1abd09159ac6.slice/crio-645a54b31898d166c52cd777cceb20b2d71f3a097b3ae926b68419c8b26f7853 WatchSource:0}: Error finding container 645a54b31898d166c52cd777cceb20b2d71f3a097b3ae926b68419c8b26f7853: Status 404 returned error can't find the container with id 645a54b31898d166c52cd777cceb20b2d71f3a097b3ae926b68419c8b26f7853 Nov 01 06:10:02.100591 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.100568 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mkk52"] Nov 01 06:10:02.284147 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.283988 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78b77c8f77-457bj"] Nov 01 06:10:02.286724 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:10:02.286696 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc06a215_884f_40b1_90fc_208bcc319a76.slice/crio-a85464f0496656c420045b4d39ac26db02312e0060eff3401b8ebd68512484e1 WatchSource:0}: Error finding container a85464f0496656c420045b4d39ac26db02312e0060eff3401b8ebd68512484e1: Status 404 returned error can't find the container with id a85464f0496656c420045b4d39ac26db02312e0060eff3401b8ebd68512484e1 Nov 01 06:10:02.737254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.737215 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkk52" event={"ID":"4fe4277a-6d12-4b3c-aaef-1abd09159ac6","Type":"ContainerStarted","Data":"645a54b31898d166c52cd777cceb20b2d71f3a097b3ae926b68419c8b26f7853"} Nov 01 06:10:02.738388 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.738364 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" event={"ID":"8f7226f4-6e78-4172-8f28-c3225d93636f","Type":"ContainerStarted","Data":"ef103cd6b15b6fefde99f2cda62d0a64ade293c907ed59218ad5a283fda460cd"} Nov 01 06:10:02.739736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.739717 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78b77c8f77-457bj" event={"ID":"cc06a215-884f-40b1-90fc-208bcc319a76","Type":"ContainerStarted","Data":"0de48e344374aecda5447a1f304e6c4277b04550ce9cb549044be44d07634bb0"} Nov 01 06:10:02.739816 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.739755 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78b77c8f77-457bj" event={"ID":"cc06a215-884f-40b1-90fc-208bcc319a76","Type":"ContainerStarted","Data":"a85464f0496656c420045b4d39ac26db02312e0060eff3401b8ebd68512484e1"} Nov 01 06:10:02.741155 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.741131 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zf9qt" event={"ID":"7fe371b4-16fb-42ce-bdd7-5e8116bcc506","Type":"ContainerStarted","Data":"71b2df0a07f952a090e9b7166137f33df2c13a5bd88740826f5c505b302eda8d"} Nov 01 06:10:02.741221 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.741176 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zf9qt" event={"ID":"7fe371b4-16fb-42ce-bdd7-5e8116bcc506","Type":"ContainerStarted","Data":"b4022c3df45da84d3ba73f601a6f9af12e0994985ec3b11623d422721581ce22"} Nov 01 06:10:02.921019 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.920988 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:02.924343 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.924320 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:02.931878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:02.931861 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l576t" Nov 01 06:10:03.748487 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:03.748459 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zf9qt" event={"ID":"7fe371b4-16fb-42ce-bdd7-5e8116bcc506","Type":"ContainerStarted","Data":"c3d18bfa37322f060ef3127c87127179ca2a8830f26bbf47f431ca26e4aac2c4"} Nov 01 06:10:03.748872 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:03.748852 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:03.750325 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:03.750302 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78b77c8f77-457bj" Nov 01 06:10:04.753850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:04.753812 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-82lzc" event={"ID":"8f7226f4-6e78-4172-8f28-c3225d93636f","Type":"ContainerStarted","Data":"e93ec39cf9a529ccc4672234a107f24368783aac9b8f9e2b17ebbc60ce5b7c39"} Nov 01 06:10:04.756579 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:04.756547 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkk52" event={"ID":"4fe4277a-6d12-4b3c-aaef-1abd09159ac6","Type":"ContainerStarted","Data":"9f0bf03dd63ddef7bd13cdcac06fef5635cf8908235fc761718db1f2dc409b24"} Nov 01 06:10:04.756712 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:04.756604 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkk52" event={"ID":"4fe4277a-6d12-4b3c-aaef-1abd09159ac6","Type":"ContainerStarted","Data":"524f15c6f0473864a169b2ed348ab0f65fd9fb11e71be5f9574d5a21738322a0"} Nov 01 06:10:05.762219 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:05.762184 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zf9qt" event={"ID":"7fe371b4-16fb-42ce-bdd7-5e8116bcc506","Type":"ContainerStarted","Data":"51ffe1e6e199dd2e36add4fc81d0ac66549924f37bceb8f7953a27cf749af200"} Nov 01 06:10:05.762660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:05.762648 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mkk52" Nov 01 06:10:07.590457 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:10:07.590427 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:10:10.317489 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:10.317457 4914 patch_prober.go:29] interesting pod/image-registry-698cf65c6b-wb9t6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 06:10:10.317918 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:10.317507 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" podUID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:10:11.001412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.001379 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x5cb4"] Nov 01 06:10:11.006189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.006169 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kx8mt" Nov 01 06:10:11.006345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.006259 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082566 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-wtmp\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082594 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-tls\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082615 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082663 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-metrics-client-ca\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082752 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082778 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-sys\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082798 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-root\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.082977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082912 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-textfile\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.083008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.082976 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlpl\" (UniqueName: \"kubernetes.io/projected/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-kube-api-access-2mlpl\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.107688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.107665 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 06:10:11.107771 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.107719 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 06:10:11.149198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.149174 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 06:10:11.149355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.149212 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 06:10:11.149355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.149269 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qh54x\"" Nov 01 06:10:11.183516 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183488 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-root\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183544 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-textfile\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183571 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlpl\" (UniqueName: \"kubernetes.io/projected/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-kube-api-access-2mlpl\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183587 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-root\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183607 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-wtmp\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183635 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-tls\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183663 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183692 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-metrics-client-ca\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183721 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183766 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-sys\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.183830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183820 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-wtmp\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.184027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183836 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-sys\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.184027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.183911 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-textfile\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.184414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.184350 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.184414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.184351 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-metrics-client-ca\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.186131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.186111 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.186309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.186289 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-node-exporter-tls\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.242261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.242242 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlpl\" (UniqueName: \"kubernetes.io/projected/ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9-kube-api-access-2mlpl\") pod \"node-exporter-x5cb4\" (UID: \"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9\") " pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.321480 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.321431 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x5cb4" Nov 01 06:10:11.333738 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:10:11.333717 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5cbfb1_5a87_4d96_b344_6f5aab9f37b9.slice/crio-ab18f3214ad2fa6cceb08aa58eeb9c5111930505de808f5010210c5c8baf928f WatchSource:0}: Error finding container ab18f3214ad2fa6cceb08aa58eeb9c5111930505de808f5010210c5c8baf928f: Status 404 returned error can't find the container with id ab18f3214ad2fa6cceb08aa58eeb9c5111930505de808f5010210c5c8baf928f Nov 01 06:10:11.781911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:11.781884 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5cb4" event={"ID":"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9","Type":"ContainerStarted","Data":"ab18f3214ad2fa6cceb08aa58eeb9c5111930505de808f5010210c5c8baf928f"} Nov 01 06:10:13.130043 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:13.130013 4914 patch_prober.go:29] interesting pod/image-registry-5d4d66c7f9-2g8gv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 01 06:10:13.130417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:13.130064 4914 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" podUID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:10:13.792022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:13.791964 4914 generic.go:356] "Generic (PLEG): container finished" podID="ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9" containerID="f5b59a31311a84e36e5ea7760cd8ff42c18c43d6884b8515151af8d6aa51ed52" exitCode=0 Nov 01 06:10:13.792135 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:13.792054 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5cb4" event={"ID":"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9","Type":"ContainerDied","Data":"f5b59a31311a84e36e5ea7760cd8ff42c18c43d6884b8515151af8d6aa51ed52"} Nov 01 06:10:14.797031 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:14.796988 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5cb4" event={"ID":"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9","Type":"ContainerStarted","Data":"5115d5c60107c6d5f4ca30adc5141b2cf43852936028b7b8a39bad9870d3c9bc"} Nov 01 06:10:14.797383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:14.797055 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x5cb4" event={"ID":"ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9","Type":"ContainerStarted","Data":"7b9c5da357ae849fd99a8753c7c4c5fdd44e1c56f2e477364448450238d86d32"} Nov 01 06:10:14.962516 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:14.962469 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x5cb4" podStartSLOduration=2.716013251 podStartE2EDuration="4.962454266s" totalImagesPullingTime="2.159916945s" totalInitContainerRuntime="86.52407ms" isStatefulPod=true podCreationTimestamp="2026-11-01 06:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:10:14.96134167 +0000 UTC m=+30.514130863" watchObservedRunningTime="2026-11-01 06:10:14.962454266 +0000 UTC m=+30.515243468" Nov 01 06:10:15.715631 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:15.715602 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:10:15.768157 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:15.768134 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mkk52" Nov 01 06:10:20.315758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:20.315729 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:10:23.684821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:23.684787 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4d66c7f9-2g8gv"] Nov 01 06:10:25.330162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.330113 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" podUID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" containerName="registry" containerID="cri-o://6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43" gracePeriod=30 Nov 01 06:10:25.567960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.567936 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:10:25.686341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686311 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686359 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686398 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686442 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686630 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686741 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686685 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686741 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686724 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686778 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") pod \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\" (UID: \"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea\") " Nov 01 06:10:25.686907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686823 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:25.687012 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.686989 4914 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-trusted-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.687130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.687094 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:25.689021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.688994 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "kube-api-access-2zt8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:25.689123 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.689038 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:25.689416 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.689385 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:25.689589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.689559 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:25.689688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.689641 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:25.693888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.693865 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted" pod "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" (UID: "ac8f514b-f92b-4e2c-b097-6594f9c2c7ea"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:25.787724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787697 4914 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-image-registry-private-configuration\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.787724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787722 4914 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-certificates\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.787903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787734 4914 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-registry-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.787903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787743 4914 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-bound-sa-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.787903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787752 4914 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-ca-trust-extracted\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.787903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787760 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zt8l\" (UniqueName: \"kubernetes.io/projected/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-kube-api-access-2zt8l\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.787903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.787769 4914 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea-installation-pull-secrets\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:25.840250 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.840223 4914 generic.go:356] "Generic (PLEG): container finished" podID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" containerID="6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43" exitCode=0 Nov 01 06:10:25.840404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.840265 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" Nov 01 06:10:25.840404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.840306 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" event={"ID":"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea","Type":"ContainerDied","Data":"6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43"} Nov 01 06:10:25.840404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.840351 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf65c6b-wb9t6" event={"ID":"ac8f514b-f92b-4e2c-b097-6594f9c2c7ea","Type":"ContainerDied","Data":"d3322e6e98647f909d3782c4402b402480558aaa45f430bc2683d7c572852261"} Nov 01 06:10:25.840404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.840372 4914 scope.go:122] "RemoveContainer" containerID="6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43" Nov 01 06:10:25.848488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.848458 4914 scope.go:122] "RemoveContainer" containerID="6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43" Nov 01 06:10:25.848747 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:10:25.848731 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43\": container with ID starting with 6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43 not found: ID does not exist" containerID="6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43" Nov 01 06:10:25.848806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:25.848753 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43"} err="failed to get container status \"6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43\": rpc error: code = NotFound desc = could not find container \"6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43\": container with ID starting with 6be0842a2c83700f58d69e3aaf6ddaece454f50a8871c40e8b9ee75a35645a43 not found: ID does not exist" Nov 01 06:10:26.003513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:26.003444 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:10:26.007779 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:26.007744 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-698cf65c6b-wb9t6"] Nov 01 06:10:26.613355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:26.613329 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" path="/var/lib/kubelet/pods/ac8f514b-f92b-4e2c-b097-6594f9c2c7ea/volumes" Nov 01 06:10:37.590509 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:10:37.590475 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:10:48.703299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:48.703246 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" podUID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" containerName="registry" containerID="cri-o://0cde7846097fc9bfc860d3ba62d7657edc0d6f06756a1fa8b8031cc5012a707c" gracePeriod=30 Nov 01 06:10:48.927161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:48.927133 4914 generic.go:356] "Generic (PLEG): container finished" podID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" containerID="0cde7846097fc9bfc860d3ba62d7657edc0d6f06756a1fa8b8031cc5012a707c" exitCode=0 Nov 01 06:10:48.927306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:48.927206 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" event={"ID":"ef1e9b60-c1a2-4e84-b155-d43c5b023617","Type":"ContainerDied","Data":"0cde7846097fc9bfc860d3ba62d7657edc0d6f06756a1fa8b8031cc5012a707c"} Nov 01 06:10:48.937111 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:48.937092 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:10:49.039500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039452 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039486 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039618 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039530 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039633 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039693 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039676 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039731 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039710 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039767 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039739 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039813 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039766 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") pod \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\" (UID: \"ef1e9b60-c1a2-4e84-b155-d43c5b023617\") " Nov 01 06:10:49.039863 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.039810 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:49.040164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.040054 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:10:49.040164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.040112 4914 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-trusted-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.041993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.041962 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:49.042301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.042260 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:49.042450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.042426 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "kube-api-access-qdrqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:49.042545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.042447 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:10:49.042545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.042504 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:10:49.046598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.046578 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted" pod "ef1e9b60-c1a2-4e84-b155-d43c5b023617" (UID: "ef1e9b60-c1a2-4e84-b155-d43c5b023617"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:10:49.141193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141167 4914 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.141193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141192 4914 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ef1e9b60-c1a2-4e84-b155-d43c5b023617-ca-trust-extracted\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.141364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141206 4914 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ef1e9b60-c1a2-4e84-b155-d43c5b023617-registry-certificates\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.141364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141218 4914 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-installation-pull-secrets\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.141364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141232 4914 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ef1e9b60-c1a2-4e84-b155-d43c5b023617-image-registry-private-configuration\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.141364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141245 4914 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-bound-sa-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.141364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.141259 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdrqw\" (UniqueName: \"kubernetes.io/projected/ef1e9b60-c1a2-4e84-b155-d43c5b023617-kube-api-access-qdrqw\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:10:49.931852 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.931827 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" Nov 01 06:10:49.932257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.931825 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d4d66c7f9-2g8gv" event={"ID":"ef1e9b60-c1a2-4e84-b155-d43c5b023617","Type":"ContainerDied","Data":"f0acbf064593c229c5aa9db745d27a85c9cb2f26e2cb0472031b5caa99949b15"} Nov 01 06:10:49.932257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:49.931951 4914 scope.go:122] "RemoveContainer" containerID="0cde7846097fc9bfc860d3ba62d7657edc0d6f06756a1fa8b8031cc5012a707c" Nov 01 06:10:50.143164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:50.143143 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d4d66c7f9-2g8gv"] Nov 01 06:10:50.193297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:50.193224 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d4d66c7f9-2g8gv"] Nov 01 06:10:50.613171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:50.613146 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" path="/var/lib/kubelet/pods/ef1e9b60-c1a2-4e84-b155-d43c5b023617/volumes" Nov 01 06:10:59.965538 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:59.965510 4914 generic.go:356] "Generic (PLEG): container finished" podID="78c096fb-88d1-4f4c-813d-b2095302bf8c" containerID="762900f1bce1250ca4d3c1757985a6571a52a22be2f2dfbf2ee11d7f96a1b075" exitCode=0 Nov 01 06:10:59.965834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:59.965562 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" event={"ID":"78c096fb-88d1-4f4c-813d-b2095302bf8c","Type":"ContainerDied","Data":"762900f1bce1250ca4d3c1757985a6571a52a22be2f2dfbf2ee11d7f96a1b075"} Nov 01 06:10:59.965834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:10:59.965795 4914 scope.go:122] "RemoveContainer" containerID="762900f1bce1250ca4d3c1757985a6571a52a22be2f2dfbf2ee11d7f96a1b075" Nov 01 06:11:00.970799 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:00.970768 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jqmxn" event={"ID":"78c096fb-88d1-4f4c-813d-b2095302bf8c","Type":"ContainerStarted","Data":"571f0aae90a03b5610119a96c3339cd1d352c8816a5a34b6911c16c2eaf08c6e"} Nov 01 06:11:03.983440 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:03.983411 4914 generic.go:356] "Generic (PLEG): container finished" podID="80c8da36-c064-46e9-9712-b36bde260e3d" containerID="3149b16f7c43e704038a18f7f5fcf42a196bc881d4c1f61b1bedee5c5590d6f8" exitCode=0 Nov 01 06:11:03.983733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:03.983480 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-72qwz" event={"ID":"80c8da36-c064-46e9-9712-b36bde260e3d","Type":"ContainerDied","Data":"3149b16f7c43e704038a18f7f5fcf42a196bc881d4c1f61b1bedee5c5590d6f8"} Nov 01 06:11:03.983733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:03.983717 4914 scope.go:122] "RemoveContainer" containerID="3149b16f7c43e704038a18f7f5fcf42a196bc881d4c1f61b1bedee5c5590d6f8" Nov 01 06:11:04.988400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:04.988368 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-72qwz" event={"ID":"80c8da36-c064-46e9-9712-b36bde260e3d","Type":"ContainerStarted","Data":"34869abdd9863816b3500a00ffa19048970fb1cfc9687ccf3379dfbd4af011c1"} Nov 01 06:11:07.590155 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:11:07.590125 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:11:15.020130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:15.020105 4914 generic.go:356] "Generic (PLEG): container finished" podID="6c66683c-48af-4ccd-9179-f61a555c3055" containerID="9eea78d72801a7783d553e90c312769408b1d16644eba546c0af7762af56b6b9" exitCode=0 Nov 01 06:11:15.020479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:15.020185 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" event={"ID":"6c66683c-48af-4ccd-9179-f61a555c3055","Type":"ContainerDied","Data":"9eea78d72801a7783d553e90c312769408b1d16644eba546c0af7762af56b6b9"} Nov 01 06:11:15.020479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:15.020448 4914 scope.go:122] "RemoveContainer" containerID="9eea78d72801a7783d553e90c312769408b1d16644eba546c0af7762af56b6b9" Nov 01 06:11:16.027689 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:11:16.027655 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-js7kl" event={"ID":"6c66683c-48af-4ccd-9179-f61a555c3055","Type":"ContainerStarted","Data":"ffacad256c53687c0a90e315f1f60b0f54fab9384eea4d0f9a660890533866bd"} Nov 01 06:11:37.591090 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:11:37.591057 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:12:07.590743 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:12:07.590664 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:12:37.590431 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:12:37.590394 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:13:07.590951 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:13:07.590912 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:13:20.096552 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096521 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw"] Nov 01 06:13:20.097002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096783 4914 cpu_manager.go:414] "containerMap: removing container" podUID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" containerName="registry" Nov 01 06:13:20.097002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096794 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" containerName="registry" Nov 01 06:13:20.097002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096818 4914 cpu_manager.go:414] "containerMap: removing container" podUID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" containerName="registry" Nov 01 06:13:20.097002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096823 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" containerName="registry" Nov 01 06:13:20.097002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096882 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8f514b-f92b-4e2c-b097-6594f9c2c7ea" containerName="registry" Nov 01 06:13:20.097002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.096893 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1e9b60-c1a2-4e84-b155-d43c5b023617" containerName="registry" Nov 01 06:13:20.099709 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.099689 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.152777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.152747 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 06:13:20.170997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.170959 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw"] Nov 01 06:13:20.201454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.201419 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 06:13:20.201454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.201433 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 06:13:20.201454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.201419 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 06:13:20.201638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.201443 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7btt7\"" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.251835 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqd4b\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.251914 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.251947 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.251975 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.252039 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.252067 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.252718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.252096 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.352991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.352900 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.352991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.352938 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.352991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.352960 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.352991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.352992 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.353355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.353009 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.353355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.353025 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.353355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.353053 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqd4b\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.353751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.353722 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.355622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.355599 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.355723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.355625 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.355954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.355934 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.355991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.355949 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.410270 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.410245 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.410429 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.410406 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqd4b\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b\") pod \"istiod-openshift-gateway-7876bfcb74-sh5qw\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:20.710180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:20.710090 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:21.126816 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:21.126785 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw"] Nov 01 06:13:21.129866 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:13:21.129838 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a9cae_6f85_4e7d_b271_f938cf54bc20.slice/crio-f884c5e6180e2afe4b41d304a18daf8c947a15181b2a20dc94c1281cc1bca589 WatchSource:0}: Error finding container f884c5e6180e2afe4b41d304a18daf8c947a15181b2a20dc94c1281cc1bca589: Status 404 returned error can't find the container with id f884c5e6180e2afe4b41d304a18daf8c947a15181b2a20dc94c1281cc1bca589 Nov 01 06:13:21.444427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:21.444343 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" event={"ID":"208a9cae-6f85-4e7d-b271-f938cf54bc20","Type":"ContainerStarted","Data":"f884c5e6180e2afe4b41d304a18daf8c947a15181b2a20dc94c1281cc1bca589"} Nov 01 06:13:25.310852 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:25.310820 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:13:25.311132 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:25.310888 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:13:25.460427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:25.460381 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" event={"ID":"208a9cae-6f85-4e7d-b271-f938cf54bc20","Type":"ContainerStarted","Data":"ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339"} Nov 01 06:13:25.460614 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:25.460491 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:26.466048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:26.465971 4914 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-sh5qw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:13:26.466048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:26.466027 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" podUID="208a9cae-6f85-4e7d-b271-f938cf54bc20" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:13:29.465972 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:29.465934 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:13:29.680830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:13:29.680779 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" podStartSLOduration=5.502451531 podStartE2EDuration="9.680764541s" totalImagesPullingTime="4.17831301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:13:25.651586879 +0000 UTC m=+221.204376066" watchObservedRunningTime="2026-11-01 06:13:29.680764541 +0000 UTC m=+225.233553726" Nov 01 06:13:37.590935 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:13:37.590888 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:14:07.590521 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:14:07.590485 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:14:21.728064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.728027 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt"] Nov 01 06:14:21.731314 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.731292 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.829015 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.828975 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.829015 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.829020 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.829232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.829093 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2jp\" (UniqueName: \"kubernetes.io/projected/993399a6-f10c-4c8d-80fa-98ab86c687ea-kube-api-access-8k2jp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.829232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.829118 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/993399a6-f10c-4c8d-80fa-98ab86c687ea-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.829232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.829148 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.829232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.829177 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993399a6-f10c-4c8d-80fa-98ab86c687ea-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.909316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.909263 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 06:14:21.909316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.909305 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 06:14:21.909593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.909298 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 06:14:21.909593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.909269 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 06:14:21.909593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.909512 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 06:14:21.909593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.909591 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-qjjpq\"" Nov 01 06:14:21.929646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.929598 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2jp\" (UniqueName: \"kubernetes.io/projected/993399a6-f10c-4c8d-80fa-98ab86c687ea-kube-api-access-8k2jp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.929646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.929656 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/993399a6-f10c-4c8d-80fa-98ab86c687ea-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.929856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.929687 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.929856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.929716 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993399a6-f10c-4c8d-80fa-98ab86c687ea-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.929856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.929740 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.929856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.929764 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.930063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.930044 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/993399a6-f10c-4c8d-80fa-98ab86c687ea-manifests\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.932350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.932321 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993399a6-f10c-4c8d-80fa-98ab86c687ea-tmp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.932487 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.932472 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.932666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.932645 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.932723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.932665 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/993399a6-f10c-4c8d-80fa-98ab86c687ea-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:21.961378 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.961350 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt"] Nov 01 06:14:21.986725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:21.986668 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2jp\" (UniqueName: \"kubernetes.io/projected/993399a6-f10c-4c8d-80fa-98ab86c687ea-kube-api-access-8k2jp\") pod \"opendatahub-operator-controller-manager-5965b9bff7-vr9xt\" (UID: \"993399a6-f10c-4c8d-80fa-98ab86c687ea\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:22.045679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:22.045646 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:22.532838 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:14:22.532809 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993399a6_f10c_4c8d_80fa_98ab86c687ea.slice/crio-37b5f7d9571efb86cb0315080ffc1a70f6fe9ea8171b580b330c8c1504eb2ece WatchSource:0}: Error finding container 37b5f7d9571efb86cb0315080ffc1a70f6fe9ea8171b580b330c8c1504eb2ece: Status 404 returned error can't find the container with id 37b5f7d9571efb86cb0315080ffc1a70f6fe9ea8171b580b330c8c1504eb2ece Nov 01 06:14:22.579286 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:22.579253 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt"] Nov 01 06:14:22.665491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:22.665452 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" event={"ID":"993399a6-f10c-4c8d-80fa-98ab86c687ea","Type":"ContainerStarted","Data":"37b5f7d9571efb86cb0315080ffc1a70f6fe9ea8171b580b330c8c1504eb2ece"} Nov 01 06:14:26.688357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:26.688251 4914 generic.go:356] "Generic (PLEG): container finished" podID="993399a6-f10c-4c8d-80fa-98ab86c687ea" containerID="bba18ebdb6e408976fdff190a896f39817e770f9e23249ca9041abecf8d01a58" exitCode=0 Nov 01 06:14:26.688357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:26.688321 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" event={"ID":"993399a6-f10c-4c8d-80fa-98ab86c687ea","Type":"ContainerDied","Data":"bba18ebdb6e408976fdff190a896f39817e770f9e23249ca9041abecf8d01a58"} Nov 01 06:14:27.694716 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:27.694676 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" event={"ID":"993399a6-f10c-4c8d-80fa-98ab86c687ea","Type":"ContainerStarted","Data":"8014e829db40b43665637cc21e4852774d914d0f66bed95025385750ce15578a"} Nov 01 06:14:27.695134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:27.694770 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:27.841754 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:27.841630 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" podStartSLOduration=2.27657862 podStartE2EDuration="6.841614974s" totalImagesPullingTime="4.10776983s" totalInitContainerRuntime="457.266524ms" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:27.840312726 +0000 UTC m=+283.393101920" watchObservedRunningTime="2026-11-01 06:14:27.841614974 +0000 UTC m=+283.394404166" Nov 01 06:14:37.593887 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:14:37.593849 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:14:38.701711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:38.701679 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965b9bff7-vr9xt" Nov 01 06:14:44.624339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:44.619907 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:14:44.625713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:44.625686 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:14:53.471024 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.470990 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh"] Nov 01 06:14:53.477055 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.477032 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.483993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.483969 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b96c1-0d3f-4af1-affa-5afb60fbebef-tls-certs\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.484101 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.484002 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htv6n\" (UniqueName: \"kubernetes.io/projected/845b96c1-0d3f-4af1-affa-5afb60fbebef-kube-api-access-htv6n\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.484144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.484103 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/845b96c1-0d3f-4af1-affa-5afb60fbebef-tmp\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.565600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.565558 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh"] Nov 01 06:14:53.584177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.584130 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 06:14:53.584379 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.584353 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hhrjq\"" Nov 01 06:14:53.584379 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.584364 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 06:14:53.584528 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.584450 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b96c1-0d3f-4af1-affa-5afb60fbebef-tls-certs\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.584528 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.584481 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htv6n\" (UniqueName: \"kubernetes.io/projected/845b96c1-0d3f-4af1-affa-5afb60fbebef-kube-api-access-htv6n\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.584600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.584537 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/845b96c1-0d3f-4af1-affa-5afb60fbebef-tmp\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.586904 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.586882 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/845b96c1-0d3f-4af1-affa-5afb60fbebef-tmp\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.587171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.587154 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845b96c1-0d3f-4af1-affa-5afb60fbebef-tls-certs\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.646840 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.646811 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htv6n\" (UniqueName: \"kubernetes.io/projected/845b96c1-0d3f-4af1-affa-5afb60fbebef-kube-api-access-htv6n\") pod \"kube-auth-proxy-68cbf89484-6x6nh\" (UID: \"845b96c1-0d3f-4af1-affa-5afb60fbebef\") " pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:53.787594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:53.787496 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" Nov 01 06:14:54.167484 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:54.167452 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh"] Nov 01 06:14:54.170472 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:14:54.170445 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845b96c1_0d3f_4af1_affa_5afb60fbebef.slice/crio-f4427a6687b7e964d5e3f1132d4ed9f97223da871f8b23c67838910fb7698cb5 WatchSource:0}: Error finding container f4427a6687b7e964d5e3f1132d4ed9f97223da871f8b23c67838910fb7698cb5: Status 404 returned error can't find the container with id f4427a6687b7e964d5e3f1132d4ed9f97223da871f8b23c67838910fb7698cb5 Nov 01 06:14:54.172377 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:54.172361 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:14:54.800783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:54.800750 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" event={"ID":"845b96c1-0d3f-4af1-affa-5afb60fbebef","Type":"ContainerStarted","Data":"f4427a6687b7e964d5e3f1132d4ed9f97223da871f8b23c67838910fb7698cb5"} Nov 01 06:14:57.815814 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:57.815774 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" event={"ID":"845b96c1-0d3f-4af1-affa-5afb60fbebef","Type":"ContainerStarted","Data":"bb8875768add340f53f3b0da81d266a4323512799a2c6a7d02d4c4c7cc4a42e3"} Nov 01 06:14:57.928783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:14:57.928645 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68cbf89484-6x6nh" podStartSLOduration=2.13183571 podStartE2EDuration="4.928631171s" totalImagesPullingTime="2.796795461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:14:57.928406792 +0000 UTC m=+313.481195985" watchObservedRunningTime="2026-11-01 06:14:57.928631171 +0000 UTC m=+313.481420550" Nov 01 06:15:05.750551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.750512 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-kbvqq"] Nov 01 06:15:05.755158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.755132 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:05.766333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.766309 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlf46\" (UniqueName: \"kubernetes.io/projected/b67d59f6-5e30-4d73-bee2-3271440235b6-kube-api-access-dlf46\") pod \"cert-manager-597b95cb84-kbvqq\" (UID: \"b67d59f6-5e30-4d73-bee2-3271440235b6\") " pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:05.766440 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.766420 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b67d59f6-5e30-4d73-bee2-3271440235b6-bound-sa-token\") pod \"cert-manager-597b95cb84-kbvqq\" (UID: \"b67d59f6-5e30-4d73-bee2-3271440235b6\") " pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:05.810314 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.810265 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kbvqq"] Nov 01 06:15:05.858629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.858588 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 06:15:05.858629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.858602 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 06:15:05.858629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.858607 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-85j7w\"" Nov 01 06:15:05.867001 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.866969 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b67d59f6-5e30-4d73-bee2-3271440235b6-bound-sa-token\") pod \"cert-manager-597b95cb84-kbvqq\" (UID: \"b67d59f6-5e30-4d73-bee2-3271440235b6\") " pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:05.867148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.867015 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlf46\" (UniqueName: \"kubernetes.io/projected/b67d59f6-5e30-4d73-bee2-3271440235b6-kube-api-access-dlf46\") pod \"cert-manager-597b95cb84-kbvqq\" (UID: \"b67d59f6-5e30-4d73-bee2-3271440235b6\") " pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:05.923529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.923501 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b67d59f6-5e30-4d73-bee2-3271440235b6-bound-sa-token\") pod \"cert-manager-597b95cb84-kbvqq\" (UID: \"b67d59f6-5e30-4d73-bee2-3271440235b6\") " pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:05.923690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:05.923625 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlf46\" (UniqueName: \"kubernetes.io/projected/b67d59f6-5e30-4d73-bee2-3271440235b6-kube-api-access-dlf46\") pod \"cert-manager-597b95cb84-kbvqq\" (UID: \"b67d59f6-5e30-4d73-bee2-3271440235b6\") " pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:06.080756 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:06.080678 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kbvqq" Nov 01 06:15:06.445592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:06.445554 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kbvqq"] Nov 01 06:15:06.449402 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:15:06.449368 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb67d59f6_5e30_4d73_bee2_3271440235b6.slice/crio-5c9d64c8640a16134f5df0cd1c9a5b105c10e1b50e775797909784767661fe85 WatchSource:0}: Error finding container 5c9d64c8640a16134f5df0cd1c9a5b105c10e1b50e775797909784767661fe85: Status 404 returned error can't find the container with id 5c9d64c8640a16134f5df0cd1c9a5b105c10e1b50e775797909784767661fe85 Nov 01 06:15:06.856007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:06.855972 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kbvqq" event={"ID":"b67d59f6-5e30-4d73-bee2-3271440235b6","Type":"ContainerStarted","Data":"5c9d64c8640a16134f5df0cd1c9a5b105c10e1b50e775797909784767661fe85"} Nov 01 06:15:07.592107 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:15:07.592068 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:15:10.886485 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:10.886443 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kbvqq" event={"ID":"b67d59f6-5e30-4d73-bee2-3271440235b6","Type":"ContainerStarted","Data":"bbdc120fc3f226dbbf9976878af1f2ad9bcfd7fb1060177e4e6c52f21f2e413f"} Nov 01 06:15:10.999482 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:15:10.999416 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-kbvqq" podStartSLOduration=2.2624292329999998 podStartE2EDuration="5.999401108s" totalImagesPullingTime="3.736971875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:15:10.998774427 +0000 UTC m=+326.551563622" watchObservedRunningTime="2026-11-01 06:15:10.999401108 +0000 UTC m=+326.552190301" Nov 01 06:15:37.591866 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:15:37.591827 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:16:07.590599 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:16:07.590561 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:16:19.111624 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.111588 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z"] Nov 01 06:16:19.113896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.113873 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.161762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.161739 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.161875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.161777 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.161943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.161878 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.161981 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.161933 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.161981 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.161962 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.162079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.161986 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7w96\" (UniqueName: \"kubernetes.io/projected/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-kube-api-access-n7w96\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.162079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.162008 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.217457 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.217414 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z"] Nov 01 06:16:19.262874 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.262850 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.262987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.262884 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.262987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.262903 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.262987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.262932 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7w96\" (UniqueName: \"kubernetes.io/projected/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-kube-api-access-n7w96\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.262987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.262959 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.263180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.263008 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.263180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.263035 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.263922 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.263895 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.265711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.265688 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.266291 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.266255 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.266394 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.266269 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.266623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.266606 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.318823 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.318802 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7w96\" (UniqueName: \"kubernetes.io/projected/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-kube-api-access-n7w96\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.318930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.318855 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6r85z\" (UID: \"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.423724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.423667 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:19.843414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.843388 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z"] Nov 01 06:16:19.845553 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:16:19.845526 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe9d4c1_5b96_4ecc_8b28_a0a66b536e63.slice/crio-1e648da4ed2d7e445a226a74f79d3843cfd49ac21d575e9412c1d8443d18b3ce WatchSource:0}: Error finding container 1e648da4ed2d7e445a226a74f79d3843cfd49ac21d575e9412c1d8443d18b3ce: Status 404 returned error can't find the container with id 1e648da4ed2d7e445a226a74f79d3843cfd49ac21d575e9412c1d8443d18b3ce Nov 01 06:16:19.847715 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.847685 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:16:19.847786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:19.847744 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:16:20.157571 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:20.157541 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" event={"ID":"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63","Type":"ContainerStarted","Data":"bb1fc7ff4ac0e16a5f35f6941cde78e648d9f83deee2f238865ad5d417fd6322"} Nov 01 06:16:20.157922 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:20.157590 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" event={"ID":"fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63","Type":"ContainerStarted","Data":"1e648da4ed2d7e445a226a74f79d3843cfd49ac21d575e9412c1d8443d18b3ce"} Nov 01 06:16:20.157922 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:20.157747 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:20.158996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:20.158972 4914 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6r85z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:16:20.159108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:20.159021 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" podUID="fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:16:20.275546 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:20.275499 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" podStartSLOduration=1.27548522 podStartE2EDuration="1.27548522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:20.274634545 +0000 UTC m=+395.827423738" watchObservedRunningTime="2026-11-01 06:16:20.27548522 +0000 UTC m=+395.828274411" Nov 01 06:16:21.161969 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:21.161925 4914 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6r85z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:16:21.162458 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:21.161984 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" podUID="fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:16:24.162035 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.162004 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6r85z" Nov 01 06:16:24.380752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.380714 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw"] Nov 01 06:16:24.380996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.380967 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" podUID="208a9cae-6f85-4e7d-b271-f938cf54bc20" containerName="discovery" containerID="cri-o://ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339" gracePeriod=30 Nov 01 06:16:24.619353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.619328 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:16:24.707794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.707762 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.707940 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.707806 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.707940 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.707825 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.708008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.707959 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.708049 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.708006 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.708083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.708060 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.708134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.708099 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts\") pod \"208a9cae-6f85-4e7d-b271-f938cf54bc20\" (UID: \"208a9cae-6f85-4e7d-b271-f938cf54bc20\") " Nov 01 06:16:24.708241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.708209 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:16:24.708415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.708393 4914 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-ca-configmap\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:24.710248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.710220 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "kube-api-access-bqd4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:24.710465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.710441 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:24.710660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.710641 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:16:24.710732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.710626 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:16:24.710772 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.710739 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:24.710772 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.710751 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig" pod "208a9cae-6f85-4e7d-b271-f938cf54bc20" (UID: "208a9cae-6f85-4e7d-b271-f938cf54bc20"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:16:24.809636 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.809605 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:24.809636 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.809634 4914 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-csr-dns-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:24.809636 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.809644 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqd4b\" (UniqueName: \"kubernetes.io/projected/208a9cae-6f85-4e7d-b271-f938cf54bc20-kube-api-access-bqd4b\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:24.809928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.809654 4914 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/208a9cae-6f85-4e7d-b271-f938cf54bc20-local-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:24.809928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.809707 4914 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-istio-kubeconfig\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:24.809928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:24.809716 4914 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/208a9cae-6f85-4e7d-b271-f938cf54bc20-cacerts\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:16:25.179837 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.179804 4914 generic.go:356] "Generic (PLEG): container finished" podID="208a9cae-6f85-4e7d-b271-f938cf54bc20" containerID="ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339" exitCode=0 Nov 01 06:16:25.180267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.179858 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" Nov 01 06:16:25.180267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.179891 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" event={"ID":"208a9cae-6f85-4e7d-b271-f938cf54bc20","Type":"ContainerDied","Data":"ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339"} Nov 01 06:16:25.180267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.179948 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw" event={"ID":"208a9cae-6f85-4e7d-b271-f938cf54bc20","Type":"ContainerDied","Data":"f884c5e6180e2afe4b41d304a18daf8c947a15181b2a20dc94c1281cc1bca589"} Nov 01 06:16:25.180267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.179975 4914 scope.go:122] "RemoveContainer" containerID="ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339" Nov 01 06:16:25.188767 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.188747 4914 scope.go:122] "RemoveContainer" containerID="ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339" Nov 01 06:16:25.189004 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:16:25.188980 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339\": container with ID starting with ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339 not found: ID does not exist" containerID="ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339" Nov 01 06:16:25.189059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.189008 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339"} err="failed to get container status \"ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339\": rpc error: code = NotFound desc = could not find container \"ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339\": container with ID starting with ec7e959b359b026f1202bd624d18ae553be0fba3b46c7ee3d30fbb6d3c742339 not found: ID does not exist" Nov 01 06:16:25.344747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.344717 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw"] Nov 01 06:16:25.349783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.349760 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-sh5qw"] Nov 01 06:16:25.507685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.507614 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8p824"] Nov 01 06:16:25.507935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.507924 4914 cpu_manager.go:414] "containerMap: removing container" podUID="208a9cae-6f85-4e7d-b271-f938cf54bc20" containerName="discovery" Nov 01 06:16:25.507975 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.507937 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208a9cae-6f85-4e7d-b271-f938cf54bc20" containerName="discovery" Nov 01 06:16:25.508007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.508002 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="208a9cae-6f85-4e7d-b271-f938cf54bc20" containerName="discovery" Nov 01 06:16:25.510294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.510254 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.562826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.562797 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 06:16:25.562826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.562798 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 06:16:25.563009 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.562798 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 06:16:25.563009 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.562896 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-k5xdn\"" Nov 01 06:16:25.568301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.568260 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8p824"] Nov 01 06:16:25.615993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.615971 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e2413a-0690-4528-8f90-94f6bd824f63-tls-certs\") pod \"model-serving-api-75bd89fdfc-8p824\" (UID: \"12e2413a-0690-4528-8f90-94f6bd824f63\") " pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.616116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.616006 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqd2\" (UniqueName: \"kubernetes.io/projected/12e2413a-0690-4528-8f90-94f6bd824f63-kube-api-access-wnqd2\") pod \"model-serving-api-75bd89fdfc-8p824\" (UID: \"12e2413a-0690-4528-8f90-94f6bd824f63\") " pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.717339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.717308 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e2413a-0690-4528-8f90-94f6bd824f63-tls-certs\") pod \"model-serving-api-75bd89fdfc-8p824\" (UID: \"12e2413a-0690-4528-8f90-94f6bd824f63\") " pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.717468 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.717359 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqd2\" (UniqueName: \"kubernetes.io/projected/12e2413a-0690-4528-8f90-94f6bd824f63-kube-api-access-wnqd2\") pod \"model-serving-api-75bd89fdfc-8p824\" (UID: \"12e2413a-0690-4528-8f90-94f6bd824f63\") " pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.720032 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.720009 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e2413a-0690-4528-8f90-94f6bd824f63-tls-certs\") pod \"model-serving-api-75bd89fdfc-8p824\" (UID: \"12e2413a-0690-4528-8f90-94f6bd824f63\") " pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.773936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.773880 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqd2\" (UniqueName: \"kubernetes.io/projected/12e2413a-0690-4528-8f90-94f6bd824f63-kube-api-access-wnqd2\") pod \"model-serving-api-75bd89fdfc-8p824\" (UID: \"12e2413a-0690-4528-8f90-94f6bd824f63\") " pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:25.821702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:25.821673 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:26.193592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:26.193569 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8p824"] Nov 01 06:16:26.196325 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:16:26.196300 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e2413a_0690_4528_8f90_94f6bd824f63.slice/crio-2d1c10c4ca5bf729e5c4fb111610dc02948d1fcb034463079efc1fd75a6ed9d1 WatchSource:0}: Error finding container 2d1c10c4ca5bf729e5c4fb111610dc02948d1fcb034463079efc1fd75a6ed9d1: Status 404 returned error can't find the container with id 2d1c10c4ca5bf729e5c4fb111610dc02948d1fcb034463079efc1fd75a6ed9d1 Nov 01 06:16:26.614436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:26.614405 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208a9cae-6f85-4e7d-b271-f938cf54bc20" path="/var/lib/kubelet/pods/208a9cae-6f85-4e7d-b271-f938cf54bc20/volumes" Nov 01 06:16:27.190258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:27.190223 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" event={"ID":"12e2413a-0690-4528-8f90-94f6bd824f63","Type":"ContainerStarted","Data":"2d1c10c4ca5bf729e5c4fb111610dc02948d1fcb034463079efc1fd75a6ed9d1"} Nov 01 06:16:29.199565 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:29.199530 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" event={"ID":"12e2413a-0690-4528-8f90-94f6bd824f63","Type":"ContainerStarted","Data":"79cad9069a2cc897cbbc6b80227b91f99c3f59b83dd166d1f18faf349bee5239"} Nov 01 06:16:29.199946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:29.199646 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:29.313095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:29.313042 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" podStartSLOduration=2.244924299 podStartE2EDuration="4.313027538s" totalImagesPullingTime="2.068103239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:29.311820235 +0000 UTC m=+404.864609430" watchObservedRunningTime="2026-11-01 06:16:29.313027538 +0000 UTC m=+404.865816730" Nov 01 06:16:37.591320 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:16:37.591256 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:16:40.207702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:40.207674 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-8p824" Nov 01 06:16:41.110260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.110228 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6"] Nov 01 06:16:41.115743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.115721 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.219406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.219377 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-csmhq\"" Nov 01 06:16:41.219749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.219439 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 06:16:41.219749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.219451 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 06:16:41.219749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.219613 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 06:16:41.219749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.219637 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 06:16:41.220348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.220330 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6"] Nov 01 06:16:41.245913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.245892 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.246017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.245930 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.246017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.245965 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28gk\" (UniqueName: \"kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.246115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.246048 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.347152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.347130 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.347301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.347173 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.347301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.347219 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28gk\" (UniqueName: \"kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.347301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.347246 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.347905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.347886 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.349824 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.349807 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.349881 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.349816 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.403161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.403111 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28gk\" (UniqueName: \"kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk\") pod \"maas-api-74d96f6f5f-5xvt6\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.428411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.428388 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:41.795474 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:41.795396 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6"] Nov 01 06:16:41.798203 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:16:41.798173 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdf3969_7a5a_443f_962c_586752671271.slice/crio-a5e94a5dbb312a08849de6f941ffc929b8f4c2ff65d6fdf7d0a5d4501263b380 WatchSource:0}: Error finding container a5e94a5dbb312a08849de6f941ffc929b8f4c2ff65d6fdf7d0a5d4501263b380: Status 404 returned error can't find the container with id a5e94a5dbb312a08849de6f941ffc929b8f4c2ff65d6fdf7d0a5d4501263b380 Nov 01 06:16:42.253697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:42.253658 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" event={"ID":"2fdf3969-7a5a-443f-962c-586752671271","Type":"ContainerStarted","Data":"a5e94a5dbb312a08849de6f941ffc929b8f4c2ff65d6fdf7d0a5d4501263b380"} Nov 01 06:16:48.287391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:48.287353 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" event={"ID":"2fdf3969-7a5a-443f-962c-586752671271","Type":"ContainerStarted","Data":"5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116"} Nov 01 06:16:48.287767 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:48.287492 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:54.297345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:54.297312 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:16:54.411212 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:54.411163 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" podStartSLOduration=7.04633411 podStartE2EDuration="13.41114976s" totalImagesPullingTime="6.36481565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:48.438945053 +0000 UTC m=+423.991734247" watchObservedRunningTime="2026-11-01 06:16:54.41114976 +0000 UTC m=+429.963938952" Nov 01 06:16:55.322450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.322418 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-664qq"] Nov 01 06:16:55.325651 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.325632 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.429049 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.429018 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-664qq"] Nov 01 06:16:55.454196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.454157 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.454196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.454197 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5tt\" (UniqueName: \"kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.454441 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.454321 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.454441 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.454364 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.555295 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.555237 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.555462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.555303 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.555462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.555353 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.555462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.555370 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5tt\" (UniqueName: \"kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.556068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.556039 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.557867 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.557846 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.557936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.557885 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.611455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.611421 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5tt\" (UniqueName: \"kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt\") pod \"maas-api-bb4d85776-664qq\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:55.637411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:55.637358 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:56.014200 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:56.014127 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-664qq"] Nov 01 06:16:56.017408 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:16:56.017377 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686b44ef_2f6c_4fe8_a62e_5d2d8e45f999.slice/crio-202d5870355886f4995e4c258430b95ca3fc293b303c2738ae2220a74dc4aaa5 WatchSource:0}: Error finding container 202d5870355886f4995e4c258430b95ca3fc293b303c2738ae2220a74dc4aaa5: Status 404 returned error can't find the container with id 202d5870355886f4995e4c258430b95ca3fc293b303c2738ae2220a74dc4aaa5 Nov 01 06:16:56.320637 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:56.320547 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" event={"ID":"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999","Type":"ContainerStarted","Data":"202d5870355886f4995e4c258430b95ca3fc293b303c2738ae2220a74dc4aaa5"} Nov 01 06:16:57.326013 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:57.325978 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" event={"ID":"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999","Type":"ContainerStarted","Data":"5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88"} Nov 01 06:16:57.326492 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:57.326065 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:16:57.439344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:16:57.439268 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" podStartSLOduration=1.631621255 podStartE2EDuration="2.439253836s" totalImagesPullingTime="807.632581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:16:57.439157229 +0000 UTC m=+432.991946422" watchObservedRunningTime="2026-11-01 06:16:57.439253836 +0000 UTC m=+432.992043029" Nov 01 06:17:03.336219 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.336177 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 06:17:03.500259 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.500221 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6"] Nov 01 06:17:03.500520 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.500482 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" podUID="2fdf3969-7a5a-443f-962c-586752671271" containerName="maas-api" containerID="cri-o://5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116" gracePeriod=30 Nov 01 06:17:03.739267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.739240 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:17:03.829667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.829638 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca\") pod \"2fdf3969-7a5a-443f-962c-586752671271\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " Nov 01 06:17:03.829842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.829681 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs\") pod \"2fdf3969-7a5a-443f-962c-586752671271\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " Nov 01 06:17:03.829842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.829703 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls\") pod \"2fdf3969-7a5a-443f-962c-586752671271\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " Nov 01 06:17:03.829842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.829740 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk\" (UniqueName: \"kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk\") pod \"2fdf3969-7a5a-443f-962c-586752671271\" (UID: \"2fdf3969-7a5a-443f-962c-586752671271\") " Nov 01 06:17:03.830033 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.830012 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca" pod "2fdf3969-7a5a-443f-962c-586752671271" (UID: "2fdf3969-7a5a-443f-962c-586752671271"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:17:03.832124 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.832086 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls" pod "2fdf3969-7a5a-443f-962c-586752671271" (UID: "2fdf3969-7a5a-443f-962c-586752671271"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:17:03.832124 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.832096 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs" pod "2fdf3969-7a5a-443f-962c-586752671271" (UID: "2fdf3969-7a5a-443f-962c-586752671271"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:17:03.832296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.832155 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk" pod "2fdf3969-7a5a-443f-962c-586752671271" (UID: "2fdf3969-7a5a-443f-962c-586752671271"). InnerVolumeSpecName "kube-api-access-w28gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:17:03.930946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.930852 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fdf3969-7a5a-443f-962c-586752671271-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:03.930946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.930887 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:03.930946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.930902 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fdf3969-7a5a-443f-962c-586752671271-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:03.930946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:03.930914 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28gk\" (UniqueName: \"kubernetes.io/projected/2fdf3969-7a5a-443f-962c-586752671271-kube-api-access-w28gk\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:17:04.356590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.356558 4914 generic.go:356] "Generic (PLEG): container finished" podID="2fdf3969-7a5a-443f-962c-586752671271" containerID="5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116" exitCode=0 Nov 01 06:17:04.357097 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.356602 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" Nov 01 06:17:04.357097 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.356612 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" event={"ID":"2fdf3969-7a5a-443f-962c-586752671271","Type":"ContainerDied","Data":"5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116"} Nov 01 06:17:04.357097 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.356652 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6" event={"ID":"2fdf3969-7a5a-443f-962c-586752671271","Type":"ContainerDied","Data":"a5e94a5dbb312a08849de6f941ffc929b8f4c2ff65d6fdf7d0a5d4501263b380"} Nov 01 06:17:04.357097 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.356672 4914 scope.go:122] "RemoveContainer" containerID="5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116" Nov 01 06:17:04.365783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.365767 4914 scope.go:122] "RemoveContainer" containerID="5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116" Nov 01 06:17:04.366039 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:17:04.366023 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116\": container with ID starting with 5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116 not found: ID does not exist" containerID="5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116" Nov 01 06:17:04.366075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.366048 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116"} err="failed to get container status \"5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116\": rpc error: code = NotFound desc = could not find container \"5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116\": container with ID starting with 5f701b1a452485e91ae8710ae30e637ee94b309314ce98e467e819dd2e9bb116 not found: ID does not exist" Nov 01 06:17:04.517599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.517564 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6"] Nov 01 06:17:04.567100 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.567065 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d96f6f5f-5xvt6"] Nov 01 06:17:04.614342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:17:04.614232 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fdf3969-7a5a-443f-962c-586752671271" path="/var/lib/kubelet/pods/2fdf3969-7a5a-443f-962c-586752671271/volumes" Nov 01 06:17:07.592377 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:17:07.592340 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:17:37.591828 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:17:37.591787 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:18:07.591579 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:18:07.591493 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:18:16.448367 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.448332 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-87wcp"] Nov 01 06:18:16.449778 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.449753 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2fdf3969-7a5a-443f-962c-586752671271" containerName="maas-api" Nov 01 06:18:16.449930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.449918 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdf3969-7a5a-443f-962c-586752671271" containerName="maas-api" Nov 01 06:18:16.450164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.450152 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdf3969-7a5a-443f-962c-586752671271" containerName="maas-api" Nov 01 06:18:16.455006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.454979 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.507846 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.507816 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-87wcp"] Nov 01 06:18:16.555928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.555899 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 06:18:16.555928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.555900 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 06:18:16.556127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.556100 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-4ktzq\"" Nov 01 06:18:16.588954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.588925 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52f8\" (UniqueName: \"kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8\") pod \"ai-gateway-controller-7d85675f98-87wcp\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.589082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.588957 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-87wcp\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.689674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.689638 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-87wcp\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.689829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.689737 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52f8\" (UniqueName: \"kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8\") pod \"ai-gateway-controller-7d85675f98-87wcp\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.692213 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.692187 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-87wcp\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.745709 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.745642 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52f8\" (UniqueName: \"kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8\") pod \"ai-gateway-controller-7d85675f98-87wcp\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:16.767490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:16.767466 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:17.131947 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:17.131732 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-87wcp"] Nov 01 06:18:17.134823 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:18:17.134790 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a95afa2_f66d_4661_841d_c626ddacd4e1.slice/crio-930fa24ab369fc6e2bf8c4bb2766bb7214f5e9a86e94f9d0fd804d47b9858b2c WatchSource:0}: Error finding container 930fa24ab369fc6e2bf8c4bb2766bb7214f5e9a86e94f9d0fd804d47b9858b2c: Status 404 returned error can't find the container with id 930fa24ab369fc6e2bf8c4bb2766bb7214f5e9a86e94f9d0fd804d47b9858b2c Nov 01 06:18:17.642134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:17.642099 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" event={"ID":"4a95afa2-f66d-4661-841d-c626ddacd4e1","Type":"ContainerStarted","Data":"930fa24ab369fc6e2bf8c4bb2766bb7214f5e9a86e94f9d0fd804d47b9858b2c"} Nov 01 06:18:19.652586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:19.652548 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" event={"ID":"4a95afa2-f66d-4661-841d-c626ddacd4e1","Type":"ContainerStarted","Data":"0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9"} Nov 01 06:18:19.652981 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:19.652662 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:19.765100 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:19.765051 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" podStartSLOduration=1.9836805320000002 podStartE2EDuration="3.765036978s" totalImagesPullingTime="1.781356446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:19.764182515 +0000 UTC m=+515.316971708" watchObservedRunningTime="2026-11-01 06:18:19.765036978 +0000 UTC m=+515.317826170" Nov 01 06:18:23.180178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.180136 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7dngs"] Nov 01 06:18:23.183580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.183557 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.240631 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.240601 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7dngs"] Nov 01 06:18:23.288270 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.288245 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 06:18:23.288385 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.288340 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fvv9q\"" Nov 01 06:18:23.343569 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.343544 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-q2fb6"] Nov 01 06:18:23.346765 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.346750 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.347810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.347789 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7dngs\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.347907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.347817 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnqp\" (UniqueName: \"kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp\") pod \"payload-pre-processing-7b98bb678c-7dngs\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.406062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.406033 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-q2fb6"] Nov 01 06:18:23.448755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.448698 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7dngs\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.448755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.448728 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnqp\" (UniqueName: \"kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp\") pod \"payload-pre-processing-7b98bb678c-7dngs\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.448908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.448755 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmf98\" (UniqueName: \"kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98\") pod \"payload-processing-55fbbbf9dc-q2fb6\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.448908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.448797 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-q2fb6\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.449404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.449388 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7dngs\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.505163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.505134 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnqp\" (UniqueName: \"kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp\") pod \"payload-pre-processing-7b98bb678c-7dngs\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:23.549385 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.549362 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmf98\" (UniqueName: \"kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98\") pod \"payload-processing-55fbbbf9dc-q2fb6\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.549480 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.549397 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-q2fb6\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.549980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.549963 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-q2fb6\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.605574 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.605550 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmf98\" (UniqueName: \"kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98\") pod \"payload-processing-55fbbbf9dc-q2fb6\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.658658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.658631 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:23.795473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:23.795446 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:24.024373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:24.024348 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-q2fb6"] Nov 01 06:18:24.026377 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:18:24.026348 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72e4bf9_85e3_4aff_b5ff_f98bcbe80124.slice/crio-ff9ece1f5ad6dc478e379a6c2052e7694db9963a668962c922a1c39ac2b647ba WatchSource:0}: Error finding container ff9ece1f5ad6dc478e379a6c2052e7694db9963a668962c922a1c39ac2b647ba: Status 404 returned error can't find the container with id ff9ece1f5ad6dc478e379a6c2052e7694db9963a668962c922a1c39ac2b647ba Nov 01 06:18:24.159047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:24.159020 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7dngs"] Nov 01 06:18:24.161434 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:18:24.161410 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e538ea_0732_44fd_bf96_f0e0131e0cdf.slice/crio-3e6d83225c379f187cf9d4419472fa33323e62ee748b3fd5f0f768766dcb5935 WatchSource:0}: Error finding container 3e6d83225c379f187cf9d4419472fa33323e62ee748b3fd5f0f768766dcb5935: Status 404 returned error can't find the container with id 3e6d83225c379f187cf9d4419472fa33323e62ee748b3fd5f0f768766dcb5935 Nov 01 06:18:24.673368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:24.673337 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" event={"ID":"91e538ea-0732-44fd-bf96-f0e0131e0cdf","Type":"ContainerStarted","Data":"3e6d83225c379f187cf9d4419472fa33323e62ee748b3fd5f0f768766dcb5935"} Nov 01 06:18:24.674412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:24.674386 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" event={"ID":"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124","Type":"ContainerStarted","Data":"ff9ece1f5ad6dc478e379a6c2052e7694db9963a668962c922a1c39ac2b647ba"} Nov 01 06:18:27.694292 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:27.694244 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" event={"ID":"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124","Type":"ContainerStarted","Data":"1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4"} Nov 01 06:18:27.694705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:27.694361 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:27.695680 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:27.695657 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" event={"ID":"91e538ea-0732-44fd-bf96-f0e0131e0cdf","Type":"ContainerStarted","Data":"77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c"} Nov 01 06:18:27.695803 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:27.695786 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:27.805600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:27.805551 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" podStartSLOduration=1.963747812 podStartE2EDuration="4.8055387s" totalImagesPullingTime="2.841790888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:27.805342327 +0000 UTC m=+523.358131521" watchObservedRunningTime="2026-11-01 06:18:27.8055387 +0000 UTC m=+523.358327893" Nov 01 06:18:27.918584 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:27.918543 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" podStartSLOduration=2.241685381 podStartE2EDuration="4.918526683s" totalImagesPullingTime="2.676841302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:27.916520328 +0000 UTC m=+523.469309525" watchObservedRunningTime="2026-11-01 06:18:27.918526683 +0000 UTC m=+523.471315875" Nov 01 06:18:30.658412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:30.658385 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:31.002125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.002056 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-575f85f67b-6vdks"] Nov 01 06:18:31.005335 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.005317 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.062021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.061998 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-575f85f67b-6vdks"] Nov 01 06:18:31.106739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.106717 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b6aac2-2f3b-48e1-b1b1-f0445358abe8-metrics-certs\") pod \"ai-gateway-controller-575f85f67b-6vdks\" (UID: \"52b6aac2-2f3b-48e1-b1b1-f0445358abe8\") " pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.106846 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.106791 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qf6h\" (UniqueName: \"kubernetes.io/projected/52b6aac2-2f3b-48e1-b1b1-f0445358abe8-kube-api-access-7qf6h\") pod \"ai-gateway-controller-575f85f67b-6vdks\" (UID: \"52b6aac2-2f3b-48e1-b1b1-f0445358abe8\") " pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.207629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.207604 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qf6h\" (UniqueName: \"kubernetes.io/projected/52b6aac2-2f3b-48e1-b1b1-f0445358abe8-kube-api-access-7qf6h\") pod \"ai-gateway-controller-575f85f67b-6vdks\" (UID: \"52b6aac2-2f3b-48e1-b1b1-f0445358abe8\") " pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.207737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.207643 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b6aac2-2f3b-48e1-b1b1-f0445358abe8-metrics-certs\") pod \"ai-gateway-controller-575f85f67b-6vdks\" (UID: \"52b6aac2-2f3b-48e1-b1b1-f0445358abe8\") " pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.209980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.209956 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52b6aac2-2f3b-48e1-b1b1-f0445358abe8-metrics-certs\") pod \"ai-gateway-controller-575f85f67b-6vdks\" (UID: \"52b6aac2-2f3b-48e1-b1b1-f0445358abe8\") " pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.263598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.263541 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qf6h\" (UniqueName: \"kubernetes.io/projected/52b6aac2-2f3b-48e1-b1b1-f0445358abe8-kube-api-access-7qf6h\") pod \"ai-gateway-controller-575f85f67b-6vdks\" (UID: \"52b6aac2-2f3b-48e1-b1b1-f0445358abe8\") " pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.317516 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.317492 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:31.680939 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.680854 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-575f85f67b-6vdks"] Nov 01 06:18:31.683482 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:18:31.683459 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b6aac2_2f3b_48e1_b1b1_f0445358abe8.slice/crio-ad838b759ee7530ddcb46499ee5b3f882465d9c5f41dc73fa263f9a9ebd177f2 WatchSource:0}: Error finding container ad838b759ee7530ddcb46499ee5b3f882465d9c5f41dc73fa263f9a9ebd177f2: Status 404 returned error can't find the container with id ad838b759ee7530ddcb46499ee5b3f882465d9c5f41dc73fa263f9a9ebd177f2 Nov 01 06:18:31.712475 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:31.712441 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" event={"ID":"52b6aac2-2f3b-48e1-b1b1-f0445358abe8","Type":"ContainerStarted","Data":"ad838b759ee7530ddcb46499ee5b3f882465d9c5f41dc73fa263f9a9ebd177f2"} Nov 01 06:18:32.718020 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:32.717992 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" event={"ID":"52b6aac2-2f3b-48e1-b1b1-f0445358abe8","Type":"ContainerStarted","Data":"f898ad4174f9669b39986605072a53cf45cef87dca95bdac8219baaa99bebf11"} Nov 01 06:18:32.718373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:32.718114 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:32.834080 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:32.834000 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" podStartSLOduration=1.98986562 podStartE2EDuration="2.833987511s" totalImagesPullingTime="844.121891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:32.832211872 +0000 UTC m=+528.385001091" watchObservedRunningTime="2026-11-01 06:18:32.833987511 +0000 UTC m=+528.386776728" Nov 01 06:18:37.590370 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:18:37.590338 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:18:38.703252 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:38.703228 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 06:18:38.703630 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:38.703317 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 06:18:43.723567 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:43.723537 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-575f85f67b-6vdks" Nov 01 06:18:43.886702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:43.886672 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-87wcp"] Nov 01 06:18:43.886867 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:43.886840 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" podUID="4a95afa2-f66d-4661-841d-c626ddacd4e1" containerName="manager" containerID="cri-o://0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9" gracePeriod=30 Nov 01 06:18:44.128018 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.127992 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:44.202523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.202502 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8\" (UniqueName: \"kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8\") pod \"4a95afa2-f66d-4661-841d-c626ddacd4e1\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " Nov 01 06:18:44.202625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.202544 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs\") pod \"4a95afa2-f66d-4661-841d-c626ddacd4e1\" (UID: \"4a95afa2-f66d-4661-841d-c626ddacd4e1\") " Nov 01 06:18:44.204663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.204639 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8" pod "4a95afa2-f66d-4661-841d-c626ddacd4e1" (UID: "4a95afa2-f66d-4661-841d-c626ddacd4e1"). InnerVolumeSpecName "kube-api-access-g52f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:18:44.204754 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.204684 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs" pod "4a95afa2-f66d-4661-841d-c626ddacd4e1" (UID: "4a95afa2-f66d-4661-841d-c626ddacd4e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:18:44.303622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.303567 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52f8\" (UniqueName: \"kubernetes.io/projected/4a95afa2-f66d-4661-841d-c626ddacd4e1-kube-api-access-g52f8\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:44.303622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.303592 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a95afa2-f66d-4661-841d-c626ddacd4e1-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:18:44.761139 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.761109 4914 generic.go:356] "Generic (PLEG): container finished" podID="4a95afa2-f66d-4661-841d-c626ddacd4e1" containerID="0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9" exitCode=0 Nov 01 06:18:44.761453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.761156 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" Nov 01 06:18:44.761453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.761203 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" event={"ID":"4a95afa2-f66d-4661-841d-c626ddacd4e1","Type":"ContainerDied","Data":"0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9"} Nov 01 06:18:44.761453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.761248 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-87wcp" event={"ID":"4a95afa2-f66d-4661-841d-c626ddacd4e1","Type":"ContainerDied","Data":"930fa24ab369fc6e2bf8c4bb2766bb7214f5e9a86e94f9d0fd804d47b9858b2c"} Nov 01 06:18:44.761453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.761293 4914 scope.go:122] "RemoveContainer" containerID="0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9" Nov 01 06:18:44.769868 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.769847 4914 scope.go:122] "RemoveContainer" containerID="0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9" Nov 01 06:18:44.770143 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:18:44.770115 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9\": container with ID starting with 0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9 not found: ID does not exist" containerID="0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9" Nov 01 06:18:44.770228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.770151 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9"} err="failed to get container status \"0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9\": rpc error: code = NotFound desc = could not find container \"0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9\": container with ID starting with 0984d37b6855607f6a8440fcf3b97171766746fe9c00f826f5f5a64c612482e9 not found: ID does not exist" Nov 01 06:18:44.922239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.922185 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-87wcp"] Nov 01 06:18:44.926048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:44.926024 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-87wcp"] Nov 01 06:18:46.614126 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:46.614096 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a95afa2-f66d-4661-841d-c626ddacd4e1" path="/var/lib/kubelet/pods/4a95afa2-f66d-4661-841d-c626ddacd4e1/volumes" Nov 01 06:18:55.441815 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.441781 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h"] Nov 01 06:18:55.442384 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.442265 4914 cpu_manager.go:414] "containerMap: removing container" podUID="4a95afa2-f66d-4661-841d-c626ddacd4e1" containerName="manager" Nov 01 06:18:55.442384 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.442302 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a95afa2-f66d-4661-841d-c626ddacd4e1" containerName="manager" Nov 01 06:18:55.442511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.442431 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a95afa2-f66d-4661-841d-c626ddacd4e1" containerName="manager" Nov 01 06:18:55.448319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.448297 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.474814 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.474789 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.474965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.474827 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.474965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.474846 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.474965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.474899 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.474965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.474930 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.475105 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.475035 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ccr\" (UniqueName: \"kubernetes.io/projected/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-kube-api-access-b9ccr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.549945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.549915 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 06:18:55.549945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.549928 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 06:18:55.549945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.549938 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hczhz\"" Nov 01 06:18:55.550126 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.549936 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 06:18:55.551317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.551291 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h"] Nov 01 06:18:55.575896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.575872 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ccr\" (UniqueName: \"kubernetes.io/projected/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-kube-api-access-b9ccr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.575996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.575913 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.575996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.575951 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.575996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.575974 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.576138 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.575997 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.576138 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.576019 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.576421 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.576398 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.576497 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.576409 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.576497 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.576455 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.578310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.578264 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.578579 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.578561 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.632125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.632105 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ccr\" (UniqueName: \"kubernetes.io/projected/e78c1a4d-d1ee-480c-87ee-1491bde3c1c1-kube-api-access-b9ccr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h\" (UID: \"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:55.761315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:55.761248 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:18:56.126130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:56.126073 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h"] Nov 01 06:18:56.128345 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:18:56.128270 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78c1a4d_d1ee_480c_87ee_1491bde3c1c1.slice/crio-517447f89cb33d23d230e60e0e222a7f6a0d9d0b9f9ccbb987c4e8f8feeab4d1 WatchSource:0}: Error finding container 517447f89cb33d23d230e60e0e222a7f6a0d9d0b9f9ccbb987c4e8f8feeab4d1: Status 404 returned error can't find the container with id 517447f89cb33d23d230e60e0e222a7f6a0d9d0b9f9ccbb987c4e8f8feeab4d1 Nov 01 06:18:56.808912 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:56.808881 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" event={"ID":"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1","Type":"ContainerStarted","Data":"517447f89cb33d23d230e60e0e222a7f6a0d9d0b9f9ccbb987c4e8f8feeab4d1"} Nov 01 06:18:58.818163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:58.818127 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" event={"ID":"e78c1a4d-d1ee-480c-87ee-1491bde3c1c1","Type":"ContainerStarted","Data":"17700f15c706668f007a415fb72e0d50b683860593e4c9fba39dfa37fe8853a8"} Nov 01 06:18:58.931136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:58.931082 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" podStartSLOduration=1.889718549 podStartE2EDuration="3.931069915s" totalImagesPullingTime="2.041351366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:18:58.930520513 +0000 UTC m=+554.483309706" watchObservedRunningTime="2026-11-01 06:18:58.931069915 +0000 UTC m=+554.483859107" Nov 01 06:18:59.761987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:18:59.761954 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:19:03.507058 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.507019 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9"] Nov 01 06:19:03.510591 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.510569 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.525006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.524984 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77908dca-3be0-4063-b49e-21b1fff1562c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.525112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.525013 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb974\" (UniqueName: \"kubernetes.io/projected/77908dca-3be0-4063-b49e-21b1fff1562c-kube-api-access-pb974\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.525112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.525032 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.525112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.525101 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.525206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.525148 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.525206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.525201 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.611963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.611935 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 06:19:03.613303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.613283 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9"] Nov 01 06:19:03.625561 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.625537 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.625665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.625596 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77908dca-3be0-4063-b49e-21b1fff1562c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.625665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.625627 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb974\" (UniqueName: \"kubernetes.io/projected/77908dca-3be0-4063-b49e-21b1fff1562c-kube-api-access-pb974\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.625665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.625654 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.625835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.625693 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.625835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.625740 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.626232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.626203 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.626380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.626358 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.626571 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.626549 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.628148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.628124 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77908dca-3be0-4063-b49e-21b1fff1562c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.628417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.628401 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77908dca-3be0-4063-b49e-21b1fff1562c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.682714 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.682695 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb974\" (UniqueName: \"kubernetes.io/projected/77908dca-3be0-4063-b49e-21b1fff1562c-kube-api-access-pb974\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gzgw9\" (UID: \"77908dca-3be0-4063-b49e-21b1fff1562c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:03.823070 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:03.823017 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:04.191142 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:04.191030 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9"] Nov 01 06:19:04.847345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:04.847316 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" event={"ID":"77908dca-3be0-4063-b49e-21b1fff1562c","Type":"ContainerStarted","Data":"80863a080bd1eec1ae6a75c540b9cb5d3c384c1b324070aec73762995e2d8f12"} Nov 01 06:19:05.761478 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:05.761444 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:19:05.773240 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:05.773222 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:19:05.853783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:05.853755 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" event={"ID":"77908dca-3be0-4063-b49e-21b1fff1562c","Type":"ContainerStarted","Data":"89b91c009e4a22e4a98037542d061860a605ed9aa7ee63e8d948fdc2a79119ea"} Nov 01 06:19:05.864334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:05.864315 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h" Nov 01 06:19:06.005086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.005045 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" podStartSLOduration=2.430059627 podStartE2EDuration="3.005033433s" totalImagesPullingTime="574.973806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:06.002920474 +0000 UTC m=+561.555709665" watchObservedRunningTime="2026-11-01 06:19:06.005033433 +0000 UTC m=+561.557822626" Nov 01 06:19:06.812148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.812114 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9"] Nov 01 06:19:06.815508 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.815491 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.824101 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.824076 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:06.850599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.850576 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.850677 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.850603 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7l5x\" (UniqueName: \"kubernetes.io/projected/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-kube-api-access-f7l5x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.850677 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.850631 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.850760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.850741 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.850794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.850787 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.850828 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.850812 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.916763 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.916738 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 06:19:06.918307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.918286 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9"] Nov 01 06:19:06.951646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.951627 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.951737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.951651 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7l5x\" (UniqueName: \"kubernetes.io/projected/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-kube-api-access-f7l5x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.951737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.951680 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.951737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.951704 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.951960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.951938 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.952028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.952002 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.952143 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.952123 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.952233 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.952214 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.952576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.952518 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.953918 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.953890 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:06.954237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:06.954219 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:07.008496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:07.008468 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7l5x\" (UniqueName: \"kubernetes.io/projected/cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef-kube-api-access-f7l5x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pmrj9\" (UID: \"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:07.126956 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:07.126932 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:07.492481 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:07.492408 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9"] Nov 01 06:19:07.495303 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:19:07.495263 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9ae92a_f1fa_4a1b_aa07_9e0ebd8113ef.slice/crio-f73545396f6ac651139b9b7650b6e48356064e82dfcc1f78636e17ebd33f599c WatchSource:0}: Error finding container f73545396f6ac651139b9b7650b6e48356064e82dfcc1f78636e17ebd33f599c: Status 404 returned error can't find the container with id f73545396f6ac651139b9b7650b6e48356064e82dfcc1f78636e17ebd33f599c Nov 01 06:19:07.590442 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:19:07.590414 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:19:07.864158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:07.864129 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" event={"ID":"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef","Type":"ContainerStarted","Data":"f73545396f6ac651139b9b7650b6e48356064e82dfcc1f78636e17ebd33f599c"} Nov 01 06:19:08.869345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:08.869312 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" event={"ID":"cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef","Type":"ContainerStarted","Data":"a4e7f17302367afaa01c1879f2c3df91968a1048efc3f14cc8d09594ee2b1516"} Nov 01 06:19:08.981815 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:08.981763 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" podStartSLOduration=2.404120537 podStartE2EDuration="2.981751741s" totalImagesPullingTime="577.631204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:08.981509082 +0000 UTC m=+564.534298274" watchObservedRunningTime="2026-11-01 06:19:08.981751741 +0000 UTC m=+564.534540933" Nov 01 06:19:09.127622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:09.127548 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:13.823940 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:13.823916 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:13.835616 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:13.835592 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:13.899419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:13.899394 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gzgw9" Nov 01 06:19:14.627363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.627330 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f"] Nov 01 06:19:14.630669 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.630655 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.710643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.710619 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.710801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.710659 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkqg\" (UniqueName: \"kubernetes.io/projected/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-kube-api-access-mvkqg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.710801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.710768 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.710801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.710792 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.710976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.710845 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.710976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.710883 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.732444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.732421 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 06:19:14.733811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.733796 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f"] Nov 01 06:19:14.812488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.811830 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.812488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.811884 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.812488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.811938 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.812488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.811982 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkqg\" (UniqueName: \"kubernetes.io/projected/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-kube-api-access-mvkqg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.812488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.812020 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.812488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.812040 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.813230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.813210 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.814788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.814763 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.815780 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.815751 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.818444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.818421 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.818761 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.818714 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.871447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.871428 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkqg\" (UniqueName: \"kubernetes.io/projected/24ae932e-1d7d-4e43-9e2c-270cbff00cf9-kube-api-access-mvkqg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f\" (UID: \"24ae932e-1d7d-4e43-9e2c-270cbff00cf9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:14.942686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:14.942634 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:15.307983 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:15.307964 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f"] Nov 01 06:19:15.310062 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:19:15.310033 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ae932e_1d7d_4e43_9e2c_270cbff00cf9.slice/crio-99f8a933d245d02ffb43458a3ed32acb2100e96cc7a43bb1698b25f51dccec9b WatchSource:0}: Error finding container 99f8a933d245d02ffb43458a3ed32acb2100e96cc7a43bb1698b25f51dccec9b: Status 404 returned error can't find the container with id 99f8a933d245d02ffb43458a3ed32acb2100e96cc7a43bb1698b25f51dccec9b Nov 01 06:19:15.899515 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:15.899487 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" event={"ID":"24ae932e-1d7d-4e43-9e2c-270cbff00cf9","Type":"ContainerStarted","Data":"99f8a933d245d02ffb43458a3ed32acb2100e96cc7a43bb1698b25f51dccec9b"} Nov 01 06:19:16.905229 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:16.905200 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" event={"ID":"24ae932e-1d7d-4e43-9e2c-270cbff00cf9","Type":"ContainerStarted","Data":"1aed107d0c329de51c67edfc99e38752c02d84118ad58d5856aede9e20513bc0"} Nov 01 06:19:16.942880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:16.942854 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:17.019230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:17.019190 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" podStartSLOduration=2.453945381 podStartE2EDuration="3.019177431s" totalImagesPullingTime="565.23205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:17.018035803 +0000 UTC m=+572.570824996" watchObservedRunningTime="2026-11-01 06:19:17.019177431 +0000 UTC m=+572.571966622" Nov 01 06:19:17.127195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:17.127170 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:17.139969 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:17.139943 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:17.919848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:17.919813 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pmrj9" Nov 01 06:19:24.943019 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:24.942991 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:24.954706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:24.954682 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:25.953119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:25.953055 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f" Nov 01 06:19:37.591469 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:19:37.591431 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:19:44.652511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:44.652487 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:19:44.661085 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:44.661063 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:19:45.127945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.127917 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh"] Nov 01 06:19:45.132912 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.132895 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.177682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.177656 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx"] Nov 01 06:19:45.180941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.180923 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.190602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.190586 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g6wqc\"" Nov 01 06:19:45.233833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.233808 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.233833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.233854 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.233896 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.233921 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.233957 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.233980 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234003 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234042 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234059 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234081 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234107 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234122 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234137 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd28q\" (UniqueName: \"kubernetes.io/projected/f393dcd3-3bd8-430f-aa5c-67af99441bdf-kube-api-access-cd28q\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234156 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234181 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234197 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234215 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234230 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswgg\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.234708 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.234590 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh"] Nov 01 06:19:45.292026 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.292002 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx"] Nov 01 06:19:45.335022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.334998 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335029 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335053 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335069 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335088 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335108 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335132 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335194 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335221 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335244 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335300 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335323 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335347 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd28q\" (UniqueName: \"kubernetes.io/projected/f393dcd3-3bd8-430f-aa5c-67af99441bdf-kube-api-access-cd28q\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335377 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335406 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335428 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335456 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.335494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335475 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.336177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335479 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswgg\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.336177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335518 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.336177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335588 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.336177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335860 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.336177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.335989 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.336177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.336109 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.336549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.336225 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.336549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.336262 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.336549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.336392 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.336549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.336481 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.337614 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.337592 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.337726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.337708 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.338106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.338086 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.338439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.338421 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.391451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.391403 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.391525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.391485 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd28q\" (UniqueName: \"kubernetes.io/projected/f393dcd3-3bd8-430f-aa5c-67af99441bdf-kube-api-access-cd28q\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.391636 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.391616 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393dcd3-3bd8-430f-aa5c-67af99441bdf-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-qpmzx\" (UID: \"f393dcd3-3bd8-430f-aa5c-67af99441bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.391746 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.391732 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswgg\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg\") pod \"maas-default-gateway-openshift-default-7d9897b54c-x4lgh\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.445390 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.445358 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:45.492617 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.492592 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:45.860951 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.860925 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh"] Nov 01 06:19:45.861998 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:19:45.861973 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8e3672_8cc0_478a_835e_adce15d48683.slice/crio-c4b84a5e00cce31298a18fd78c006446738e0affb0988aa8d24651567bac29de WatchSource:0}: Error finding container c4b84a5e00cce31298a18fd78c006446738e0affb0988aa8d24651567bac29de: Status 404 returned error can't find the container with id c4b84a5e00cce31298a18fd78c006446738e0affb0988aa8d24651567bac29de Nov 01 06:19:45.912778 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:45.912756 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx"] Nov 01 06:19:45.914449 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:19:45.914420 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf393dcd3_3bd8_430f_aa5c_67af99441bdf.slice/crio-c5fc84bc1cdd47997fe87d0e307900d2bff579937a596aa3c7f90b20ccf4a5dd WatchSource:0}: Error finding container c5fc84bc1cdd47997fe87d0e307900d2bff579937a596aa3c7f90b20ccf4a5dd: Status 404 returned error can't find the container with id c5fc84bc1cdd47997fe87d0e307900d2bff579937a596aa3c7f90b20ccf4a5dd Nov 01 06:19:46.030916 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:46.030886 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" event={"ID":"f393dcd3-3bd8-430f-aa5c-67af99441bdf","Type":"ContainerStarted","Data":"c5fc84bc1cdd47997fe87d0e307900d2bff579937a596aa3c7f90b20ccf4a5dd"} Nov 01 06:19:46.032124 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:46.032103 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" event={"ID":"7a8e3672-8cc0-478a-835e-adce15d48683","Type":"ContainerStarted","Data":"c4b84a5e00cce31298a18fd78c006446738e0affb0988aa8d24651567bac29de"} Nov 01 06:19:49.672661 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:49.672625 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:19:49.673044 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:49.672697 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:19:49.673044 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:49.672728 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:19:49.680687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:49.680655 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:19:49.680785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:49.680756 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:19:49.680821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:49.680798 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:19:50.055152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.055052 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" event={"ID":"f393dcd3-3bd8-430f-aa5c-67af99441bdf","Type":"ContainerStarted","Data":"d957b7628126bc9f1646fc5694534e18741ff8f9842d1ad6b126dcaabaf95877"} Nov 01 06:19:50.056746 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.056720 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" event={"ID":"7a8e3672-8cc0-478a-835e-adce15d48683","Type":"ContainerStarted","Data":"c3d70008b1e5f649597d686c51bcd072f84284ea3764c43e148ddcc70498d3d9"} Nov 01 06:19:50.219002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.218927 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podStartSLOduration=1.463462136 podStartE2EDuration="5.218909388s" totalImagesPullingTime="3.755447252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:50.218819269 +0000 UTC m=+605.771608462" watchObservedRunningTime="2026-11-01 06:19:50.218909388 +0000 UTC m=+605.771698580" Nov 01 06:19:50.334792 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.334674 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podStartSLOduration=1.518610161 podStartE2EDuration="5.334658854s" totalImagesPullingTime="3.816048693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:19:50.333542954 +0000 UTC m=+605.886332160" watchObservedRunningTime="2026-11-01 06:19:50.334658854 +0000 UTC m=+605.887448047" Nov 01 06:19:50.461301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.457350 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:50.461476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.461383 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:50.461476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.461438 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:50.494777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.494729 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:50.496113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.496085 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:50.496361 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:50.496321 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:51.446657 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:51.446621 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:51.447238 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:51.447207 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:51.494679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:51.494567 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:51.494679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:51.494616 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:52.451303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:52.449649 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:52.451303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:52.449710 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:52.494299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:52.493652 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:52.494299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:52.493711 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:53.453301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:53.449628 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:53.453301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:53.449701 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:53.495658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:53.495622 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:53.495919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:53.495892 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:54.464298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:54.453616 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:54.464298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:54.453675 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:54.497373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:54.497340 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:54.497624 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:54.497598 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:55.449724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:55.449687 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:55.450002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:55.449974 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:55.493250 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:55.493216 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:55.493620 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:55.493288 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:56.445753 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:56.445717 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-x4lgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 01 06:19:56.445912 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:56.445793 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 01 06:19:56.493341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:56.493314 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 01 06:19:56.493652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:56.493357 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 01 06:19:57.737657 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:57.737624 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:57.738103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:57.738056 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:57.738850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:57.738826 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:19:57.766815 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:57.766789 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:57.767136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:57.767116 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:19:57.767984 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:19:57.767967 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" Nov 01 06:20:07.591880 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:07.591838 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:20:24.127964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.127929 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq"] Nov 01 06:20:24.136073 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.136054 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.226442 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.226421 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8ff537-openshift-default-dockercfg-sbjxb\"" Nov 01 06:20:24.238977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.238949 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq"] Nov 01 06:20:24.266419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266388 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266431 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266449 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266467 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266702 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266780 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266757 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266793 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx7hp\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266869 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.266933 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.266892 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.367807 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.367773 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.367965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.367812 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.367965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.367829 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.367965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.367946 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368006 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368037 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368070 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx7hp\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368153 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368177 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368205 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368425 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368303 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368505 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368728 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.368886 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.368745 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.370467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.370446 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.370965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.370944 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.428308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.428234 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.428707 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.428689 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx7hp\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.434826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.434806 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8"] Nov 01 06:20:24.440073 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.440055 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.448983 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.448966 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:24.492323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.492289 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ba2f3-openshift-default-dockercfg-nrd89\"" Nov 01 06:20:24.499001 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.498968 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8"] Nov 01 06:20:24.570517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570485 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570662 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570589 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570667 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skl6\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570700 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570770 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570810 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570902 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.570966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570939 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.571020 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.570976 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.671842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.671810 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.671842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.671845 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.671865 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.671986 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672024 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672080 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672180 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672189 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672210 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672236 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672264 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skl6\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672331 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672462 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.672808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.672788 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.674493 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.674476 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.674700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.674684 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.731553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.731492 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.731643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.731611 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skl6\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.759967 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.759943 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:24.874444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.874421 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq"] Nov 01 06:20:24.877142 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:24.877117 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace0ebde_a5f8_45e1_ac9e_769a7a88daf3.slice/crio-8e1a9c98401e8a618768dcc6aa0d27ca8deb278b1dad4cea21e3022d7954b89f WatchSource:0}: Error finding container 8e1a9c98401e8a618768dcc6aa0d27ca8deb278b1dad4cea21e3022d7954b89f: Status 404 returned error can't find the container with id 8e1a9c98401e8a618768dcc6aa0d27ca8deb278b1dad4cea21e3022d7954b89f Nov 01 06:20:24.879263 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.879220 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:24.879380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.879337 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:24.879438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:24.879381 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:25.135685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.135661 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8"] Nov 01 06:20:25.138421 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:25.138400 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea528e_6627_4cc2_b4a2_73fa7f47c2f5.slice/crio-3089e41603010bdd593f8ab54829bf150963b3b970ec6a5a0ebcba10df52899f WatchSource:0}: Error finding container 3089e41603010bdd593f8ab54829bf150963b3b970ec6a5a0ebcba10df52899f: Status 404 returned error can't find the container with id 3089e41603010bdd593f8ab54829bf150963b3b970ec6a5a0ebcba10df52899f Nov 01 06:20:25.141386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.141355 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:25.141464 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.141427 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:25.141464 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.141455 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:25.241773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.241739 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" event={"ID":"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3","Type":"ContainerStarted","Data":"72f7c099b07cb14a52a0b3e4cf8e36a986156487a64d598215e821e19209cba5"} Nov 01 06:20:25.242163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.241806 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" event={"ID":"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3","Type":"ContainerStarted","Data":"8e1a9c98401e8a618768dcc6aa0d27ca8deb278b1dad4cea21e3022d7954b89f"} Nov 01 06:20:25.243622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.243597 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" event={"ID":"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5","Type":"ContainerStarted","Data":"6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00"} Nov 01 06:20:25.243722 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.243643 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" event={"ID":"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5","Type":"ContainerStarted","Data":"3089e41603010bdd593f8ab54829bf150963b3b970ec6a5a0ebcba10df52899f"} Nov 01 06:20:25.360397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.360342 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podStartSLOduration=1.360329232 podStartE2EDuration="1.360329232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:25.357672512 +0000 UTC m=+640.910461706" watchObservedRunningTime="2026-11-01 06:20:25.360329232 +0000 UTC m=+640.913118423" Nov 01 06:20:25.449573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.449502 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:25.454168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.454140 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:25.477332 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.477248 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podStartSLOduration=1.477233425 podStartE2EDuration="1.477233425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:25.473862136 +0000 UTC m=+641.026651332" watchObservedRunningTime="2026-11-01 06:20:25.477233425 +0000 UTC m=+641.030022618" Nov 01 06:20:25.760317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:25.760230 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:26.248010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:26.247980 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:26.248956 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:26.248934 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:20:26.766665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:26.766635 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:27.252319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:27.252293 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:27.253071 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:27.253052 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:31.804267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.804229 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d"] Nov 01 06:20:31.808075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.808053 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:31.869060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.869031 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8ff537\"" Nov 01 06:20:31.921582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.921548 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8ff537-dockercfg-jn9jt\"" Nov 01 06:20:31.924584 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.924554 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d"] Nov 01 06:20:31.939023 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.938998 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:31.939145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.939055 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm2l\" (UniqueName: \"kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l\") pod \"payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:31.946374 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.946353 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd"] Nov 01 06:20:31.949682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:31.949666 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.006377 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.006353 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd"] Nov 01 06:20:32.040201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.040174 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:32.040314 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.040233 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm2l\" (UniqueName: \"kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l\") pod \"payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:32.040809 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.040792 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:32.102120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.102097 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm2l\" (UniqueName: \"kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l\") pod \"payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:32.122105 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.122086 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:32.141202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.141179 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.141326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.141307 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7vg\" (UniqueName: \"kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg\") pod \"payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.241946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.241909 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7vg\" (UniqueName: \"kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg\") pod \"payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.242102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.241974 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.242714 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.242696 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.298022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.297996 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7vg\" (UniqueName: \"kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg\") pod \"payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.485673 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.485647 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d"] Nov 01 06:20:32.488069 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:32.488039 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e69806a_6277_4edc_85a7_1d0038b53e30.slice/crio-ffa63efc8a6ae6712ab389682ac4326b7909e77f5f4e27ec008dceaa6f79ec44 WatchSource:0}: Error finding container ffa63efc8a6ae6712ab389682ac4326b7909e77f5f4e27ec008dceaa6f79ec44: Status 404 returned error can't find the container with id ffa63efc8a6ae6712ab389682ac4326b7909e77f5f4e27ec008dceaa6f79ec44 Nov 01 06:20:32.563360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.563341 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:32.927693 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:32.927669 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd"] Nov 01 06:20:32.929380 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:32.929353 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6802946e_4e90_4b1a_80bb_a04b17f3b2ae.slice/crio-b513ff12a123f54f10c315f345631cd1fbe771d44377ec0e6d8ab67aa5c07f91 WatchSource:0}: Error finding container b513ff12a123f54f10c315f345631cd1fbe771d44377ec0e6d8ab67aa5c07f91: Status 404 returned error can't find the container with id b513ff12a123f54f10c315f345631cd1fbe771d44377ec0e6d8ab67aa5c07f91 Nov 01 06:20:33.280245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.280178 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" event={"ID":"8e69806a-6277-4edc-85a7-1d0038b53e30","Type":"ContainerStarted","Data":"b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6"} Nov 01 06:20:33.280245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.280240 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" event={"ID":"8e69806a-6277-4edc-85a7-1d0038b53e30","Type":"ContainerStarted","Data":"ffa63efc8a6ae6712ab389682ac4326b7909e77f5f4e27ec008dceaa6f79ec44"} Nov 01 06:20:33.280460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.280396 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:33.281785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.281763 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" event={"ID":"6802946e-4e90-4b1a-80bb-a04b17f3b2ae","Type":"ContainerStarted","Data":"5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20"} Nov 01 06:20:33.281883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.281799 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" event={"ID":"6802946e-4e90-4b1a-80bb-a04b17f3b2ae","Type":"ContainerStarted","Data":"b513ff12a123f54f10c315f345631cd1fbe771d44377ec0e6d8ab67aa5c07f91"} Nov 01 06:20:33.281883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.281831 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:33.394603 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.394561 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" podStartSLOduration=2.394550802 podStartE2EDuration="2.394550802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:33.392785023 +0000 UTC m=+648.945574215" watchObservedRunningTime="2026-11-01 06:20:33.394550802 +0000 UTC m=+648.947340035" Nov 01 06:20:33.512585 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:33.512468 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" podStartSLOduration=2.512448204 podStartE2EDuration="2.512448204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:33.509442386 +0000 UTC m=+649.062231579" watchObservedRunningTime="2026-11-01 06:20:33.512448204 +0000 UTC m=+649.065237398" Nov 01 06:20:34.463949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.463920 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx"] Nov 01 06:20:34.467524 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.467508 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.519005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.518986 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3ba2f3-serving-cert\"" Nov 01 06:20:34.570352 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.570327 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx"] Nov 01 06:20:34.661113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.661085 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tfnf\" (UniqueName: \"kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.661214 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.661120 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.661248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.661226 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.661337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.661296 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.762551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.762465 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.762551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.762520 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tfnf\" (UniqueName: \"kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.762782 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.762551 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.762782 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.762639 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.762947 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:34.762806 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-serving-cert: secret "maas-api-e2e-worker-w0-3ba2f3-serving-cert" not found Nov 01 06:20:34.762947 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:34.762884 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls podName:19fb1a00-b72d-47cc-92be-78f39f2efeeb nodeName:}" failed. No retries permitted until 2026-11-01 06:20:35.262849284 +0000 UTC m=+650.815638471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls") pod "maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" (UID: "19fb1a00-b72d-47cc-92be-78f39f2efeeb") : secret "maas-api-e2e-worker-w0-3ba2f3-serving-cert" not found Nov 01 06:20:34.763436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.763416 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.765295 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.765259 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:34.819419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:34.819393 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tfnf\" (UniqueName: \"kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:35.266398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:35.266365 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:35.268744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:35.268726 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:35.379762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:35.379738 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:35.744652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:35.744629 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx"] Nov 01 06:20:35.746432 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:35.746407 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fb1a00_b72d_47cc_92be_78f39f2efeeb.slice/crio-f47cdccb1e2be1e4bbd1dd9fbbdde20c9f557efb1d5f27a7aedc7b341d2be9d9 WatchSource:0}: Error finding container f47cdccb1e2be1e4bbd1dd9fbbdde20c9f557efb1d5f27a7aedc7b341d2be9d9: Status 404 returned error can't find the container with id f47cdccb1e2be1e4bbd1dd9fbbdde20c9f557efb1d5f27a7aedc7b341d2be9d9 Nov 01 06:20:35.747786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:35.747767 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:20:36.295516 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:36.295483 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" event={"ID":"19fb1a00-b72d-47cc-92be-78f39f2efeeb","Type":"ContainerStarted","Data":"f47cdccb1e2be1e4bbd1dd9fbbdde20c9f557efb1d5f27a7aedc7b341d2be9d9"} Nov 01 06:20:37.301784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:37.301747 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" event={"ID":"19fb1a00-b72d-47cc-92be-78f39f2efeeb","Type":"ContainerStarted","Data":"3278cee798fbf0e50627b25b2f038a9a7cf3469c503a5c1b40515810d049146e"} Nov 01 06:20:37.302175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:37.301832 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:37.591263 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:37.591184 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:20:39.291692 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.291641 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" podStartSLOduration=4.420672144 podStartE2EDuration="5.291626453s" totalImagesPullingTime="870.954309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:37.420219955 +0000 UTC m=+652.973009149" watchObservedRunningTime="2026-11-01 06:20:39.291626453 +0000 UTC m=+654.844415644" Nov 01 06:20:39.292971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.292953 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk"] Nov 01 06:20:39.296755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.296741 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.400397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.400365 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-8ff537-serving-cert\"" Nov 01 06:20:39.402398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.402377 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk"] Nov 01 06:20:39.406789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.406759 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.406882 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.406809 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.406923 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.406894 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.406952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.406932 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8t2\" (UniqueName: \"kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.507609 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.507579 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.507721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.507625 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8t2\" (UniqueName: \"kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.507721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.507705 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.507826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.507733 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.507826 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:39.507733 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-serving-cert: secret "maas-api-e2e-worker-w2-8ff537-serving-cert" not found Nov 01 06:20:39.507924 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:39.507830 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls podName:80f62e03-6c2a-4002-b6e5-a0dd06c88917 nodeName:}" failed. No retries permitted until 2026-11-01 06:20:40.007809308 +0000 UTC m=+655.560598479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls") pod "maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" (UID: "80f62e03-6c2a-4002-b6e5-a0dd06c88917") : secret "maas-api-e2e-worker-w2-8ff537-serving-cert" not found Nov 01 06:20:39.508230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.508207 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.510079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.510058 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:39.564764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:39.564695 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8t2\" (UniqueName: \"kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:40.011580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:40.011548 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:40.013936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:40.013916 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:40.208715 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:40.208696 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:40.576819 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:40.576797 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk"] Nov 01 06:20:40.578771 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:40.578746 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f62e03_6c2a_4002_b6e5_a0dd06c88917.slice/crio-21997c93d2f6985f760bd951874e9e6f1c9aaab34e7dc977ac7a477940f1cfc9 WatchSource:0}: Error finding container 21997c93d2f6985f760bd951874e9e6f1c9aaab34e7dc977ac7a477940f1cfc9: Status 404 returned error can't find the container with id 21997c93d2f6985f760bd951874e9e6f1c9aaab34e7dc977ac7a477940f1cfc9 Nov 01 06:20:41.321537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:41.321506 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" event={"ID":"80f62e03-6c2a-4002-b6e5-a0dd06c88917","Type":"ContainerStarted","Data":"21997c93d2f6985f760bd951874e9e6f1c9aaab34e7dc977ac7a477940f1cfc9"} Nov 01 06:20:42.327929 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:42.327899 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" event={"ID":"80f62e03-6c2a-4002-b6e5-a0dd06c88917","Type":"ContainerStarted","Data":"1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615"} Nov 01 06:20:42.328371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:42.327964 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:42.443134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:42.443081 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" podStartSLOduration=2.661293781 podStartE2EDuration="3.443067554s" totalImagesPullingTime="781.773773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:42.442049928 +0000 UTC m=+657.994839183" watchObservedRunningTime="2026-11-01 06:20:42.443067554 +0000 UTC m=+657.995856745" Nov 01 06:20:43.311977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:43.311948 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 06:20:44.059300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.057472 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9"] Nov 01 06:20:44.073205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.073179 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.124463 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.124424 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9"] Nov 01 06:20:44.172464 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.172440 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-190724-serving-cert\"" Nov 01 06:20:44.182935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.182905 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.183038 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.182940 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.183077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.183065 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2tb\" (UniqueName: \"kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.183110 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.183088 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.283832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.283805 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.283954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.283841 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.283954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.283903 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2tb\" (UniqueName: \"kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.283954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.283930 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.284091 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:44.283949 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-serving-cert: secret "maas-api-e2e-worker-w4-190724-serving-cert" not found Nov 01 06:20:44.284091 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:20:44.284007 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls podName:be12909e-3dac-4a7b-9fdd-dae945a16a44 nodeName:}" failed. No retries permitted until 2026-11-01 06:20:44.783990587 +0000 UTC m=+660.336779758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls") pod "maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" (UID: "be12909e-3dac-4a7b-9fdd-dae945a16a44") : secret "maas-api-e2e-worker-w4-190724-serving-cert" not found Nov 01 06:20:44.284651 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.284634 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.286391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.286375 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.288912 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.288877 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 06:20:44.289005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.288949 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 06:20:44.344047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.343985 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2tb\" (UniqueName: \"kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.788774 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.788738 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.791114 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.791094 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\") pod \"maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:44.986007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:44.985976 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:45.401993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:45.401970 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9"] Nov 01 06:20:45.404290 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:20:45.404253 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe12909e_3dac_4a7b_9fdd_dae945a16a44.slice/crio-5e1f700daef1031dab29cac386aa2088d62fc0a1e1711c8095410d7ccb736ccc WatchSource:0}: Error finding container 5e1f700daef1031dab29cac386aa2088d62fc0a1e1711c8095410d7ccb736ccc: Status 404 returned error can't find the container with id 5e1f700daef1031dab29cac386aa2088d62fc0a1e1711c8095410d7ccb736ccc Nov 01 06:20:46.371817 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:46.371778 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" event={"ID":"be12909e-3dac-4a7b-9fdd-dae945a16a44","Type":"ContainerStarted","Data":"e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509"} Nov 01 06:20:46.371989 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:46.371840 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" event={"ID":"be12909e-3dac-4a7b-9fdd-dae945a16a44","Type":"ContainerStarted","Data":"5e1f700daef1031dab29cac386aa2088d62fc0a1e1711c8095410d7ccb736ccc"} Nov 01 06:20:46.371989 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:46.371866 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:46.486148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:46.486026 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" podStartSLOduration=1.718116119 podStartE2EDuration="2.486011581s" totalImagesPullingTime="767.895462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:20:46.484100773 +0000 UTC m=+662.036889964" watchObservedRunningTime="2026-11-01 06:20:46.486011581 +0000 UTC m=+662.038800775" Nov 01 06:20:48.338988 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:48.338961 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 06:20:52.381734 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.381701 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 06:20:52.401622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.401592 4914 generic.go:356] "Generic (PLEG): container finished" podID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerID="6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00" exitCode=0 Nov 01 06:20:52.401751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.401659 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" event={"ID":"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5","Type":"ContainerDied","Data":"6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00"} Nov 01 06:20:52.401973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.401960 4914 scope.go:122] "RemoveContainer" containerID="6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00" Nov 01 06:20:52.403104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.403011 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:52.403104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.403074 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:52.403194 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:52.403109 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:20:53.408186 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:53.408151 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" event={"ID":"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5","Type":"ContainerStarted","Data":"25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1"} Nov 01 06:20:53.780360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:53.763467 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:53.780360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:53.768368 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 01 06:20:53.780360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:53.768419 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 01 06:20:54.761795 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:54.761758 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 01 06:20:54.762459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:54.762421 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 01 06:20:55.765254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:55.765215 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 01 06:20:55.766004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:55.765968 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 01 06:20:56.763004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:56.762588 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 01 06:20:56.763004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:56.762646 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 01 06:20:57.265299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:57.264387 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:57.762610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:57.762574 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 01 06:20:57.762863 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:57.762835 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 01 06:20:58.764499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:58.764463 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:20:58.765612 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:20:58.765590 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 06:21:01.448950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:01.448918 4914 generic.go:356] "Generic (PLEG): container finished" podID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerID="72f7c099b07cb14a52a0b3e4cf8e36a986156487a64d598215e821e19209cba5" exitCode=0 Nov 01 06:21:01.449382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:01.448986 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" event={"ID":"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3","Type":"ContainerDied","Data":"72f7c099b07cb14a52a0b3e4cf8e36a986156487a64d598215e821e19209cba5"} Nov 01 06:21:01.449382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:01.449373 4914 scope.go:122] "RemoveContainer" containerID="72f7c099b07cb14a52a0b3e4cf8e36a986156487a64d598215e821e19209cba5" Nov 01 06:21:01.450499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:01.450468 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:21:01.450565 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:01.450538 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:21:01.450602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:01.450591 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:21:02.474324 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:02.474256 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" event={"ID":"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3","Type":"ContainerStarted","Data":"121d7d5438e4fd927d2dd419ad7be1cb68adcdb1df5e61ad1739016ee9c3d321"} Nov 01 06:21:03.459300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:03.449597 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:21:03.459300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:03.449713 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 01 06:21:03.459300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:03.449756 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 01 06:21:04.452300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:04.452147 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 01 06:21:04.452300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:04.452207 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 01 06:21:05.454688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:05.454654 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 01 06:21:05.455394 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:05.455363 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 01 06:21:06.449659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:06.449622 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 01 06:21:06.449965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:06.449936 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 01 06:21:07.453333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:07.453303 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:21:07.453745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:07.453560 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:21:07.454249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:07.454233 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:21:07.599389 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:21:07.599356 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:21:30.335159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.335120 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd"] Nov 01 06:21:30.338903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.338881 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457442 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457613 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457486 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457613 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457509 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2mx\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457648 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457742 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457816 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457801 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457859 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457842 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457892 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457871 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.457942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.457924 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.542761 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.542732 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd"] Nov 01 06:21:30.558584 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558561 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558591 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558607 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2mx\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558627 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558681 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558714 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558748 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558775 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.558985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.558810 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.559128 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.559061 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.559184 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.559141 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.559184 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.559165 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.559486 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.559461 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.559621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.559604 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.561642 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.561591 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.561728 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.561660 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.615116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.615091 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.615401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.615381 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2mx\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx\") pod \"maas-default-gateway-openshift-default-7d9897b54c-wvfkd\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:30.650989 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:30.650968 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:31.069614 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.069587 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd"] Nov 01 06:21:31.071402 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:21:31.071377 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fa915f_666d_45ac_9b5f_839e54ac78eb.slice/crio-00b9197178a195610fb89eb2e461d3a18a123e2f26ee1f929f729c1d7567616e WatchSource:0}: Error finding container 00b9197178a195610fb89eb2e461d3a18a123e2f26ee1f929f729c1d7567616e: Status 404 returned error can't find the container with id 00b9197178a195610fb89eb2e461d3a18a123e2f26ee1f929f729c1d7567616e Nov 01 06:21:31.073678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.073639 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:21:31.073777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.073706 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:21:31.073777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.073736 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:21:31.620455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.620411 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" event={"ID":"38fa915f-666d-45ac-9b5f-839e54ac78eb","Type":"ContainerStarted","Data":"60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf"} Nov 01 06:21:31.620820 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.620468 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" event={"ID":"38fa915f-666d-45ac-9b5f-839e54ac78eb","Type":"ContainerStarted","Data":"00b9197178a195610fb89eb2e461d3a18a123e2f26ee1f929f729c1d7567616e"} Nov 01 06:21:31.672367 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.671731 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:31.785004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:31.784954 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podStartSLOduration=1.784940056 podStartE2EDuration="1.784940056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:31.783830595 +0000 UTC m=+707.336619810" watchObservedRunningTime="2026-11-01 06:21:31.784940056 +0000 UTC m=+707.337729247" Nov 01 06:21:32.667054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.666738 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-wvfkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 06:21:32.667054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.666791 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 06:21:32.774319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.774290 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m"] Nov 01 06:21:32.777836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.777821 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.829624 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.829597 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-85584b-serving-cert\"" Nov 01 06:21:32.836366 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.836335 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m"] Nov 01 06:21:32.888559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.888534 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.888684 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.888584 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.888726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.888688 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.888789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.888777 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mv87\" (UniqueName: \"kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.989616 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.989534 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.989749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.989628 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.989749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.989683 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.989865 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.989752 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mv87\" (UniqueName: \"kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.989865 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:21:32.989772 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-serving-cert: secret "maas-api-e2e-inf-a-85584b-serving-cert" not found Nov 01 06:21:32.989865 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:21:32.989832 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls podName:adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92 nodeName:}" failed. No retries permitted until 2026-11-01 06:21:33.489813841 +0000 UTC m=+709.042603031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls") pod "maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" (UID: "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92") : secret "maas-api-e2e-inf-a-85584b-serving-cert" not found Nov 01 06:21:32.990171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.990151 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:32.992166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:32.992146 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:33.073022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:33.072987 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mv87\" (UniqueName: \"kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:33.495987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:33.495946 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:33.498506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:33.498474 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\") pod \"maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:33.666674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:33.666641 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-wvfkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 06:21:33.666932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:33.666902 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 06:21:33.691283 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:33.691258 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:34.072724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:34.072699 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m"] Nov 01 06:21:34.075116 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:21:34.075086 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfc3b19_8c3a_4a9f_9b5a_953c7b3a4a92.slice/crio-6af0f845a632e24954448dd267092b8ee24e886a4341476cbba2e6a6b1248c2e WatchSource:0}: Error finding container 6af0f845a632e24954448dd267092b8ee24e886a4341476cbba2e6a6b1248c2e: Status 404 returned error can't find the container with id 6af0f845a632e24954448dd267092b8ee24e886a4341476cbba2e6a6b1248c2e Nov 01 06:21:34.635157 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:34.635124 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" event={"ID":"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92","Type":"ContainerStarted","Data":"6af0f845a632e24954448dd267092b8ee24e886a4341476cbba2e6a6b1248c2e"} Nov 01 06:21:34.666297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:34.665646 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-wvfkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 06:21:34.666297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:34.665700 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 06:21:35.641758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:35.641725 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" event={"ID":"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92","Type":"ContainerStarted","Data":"87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf"} Nov 01 06:21:35.642139 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:35.641870 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:21:35.668987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:35.668587 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-wvfkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 06:21:35.668987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:35.668643 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 06:21:36.660304 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:36.656331 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-wvfkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 06:21:36.660304 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:36.656381 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 06:21:37.591598 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:21:37.591558 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:21:37.651775 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:37.651739 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-wvfkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 01 06:21:37.651921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:37.651813 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 01 06:21:38.955640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:38.955609 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:38.956076 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:38.955967 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:38.957219 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:38.957195 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:21:39.122809 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:39.122750 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" podStartSLOduration=6.287656702 podStartE2EDuration="7.12273445s" totalImagesPullingTime="835.077748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:21:35.770426163 +0000 UTC m=+711.323215356" watchObservedRunningTime="2026-11-01 06:21:39.12273445 +0000 UTC m=+714.675523641" Nov 01 06:21:41.652602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:21:41.652578 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:22:07.026230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.026193 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw"] Nov 01 06:22:07.030240 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.030217 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.088360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.088309 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef9b55-openshift-default-dockercfg-s8qvw\"" Nov 01 06:22:07.122568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122535 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122585 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122609 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122707 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122744 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122773 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b2p5\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122894 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122881 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122923 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.122976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.122964 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.133830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.133804 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw"] Nov 01 06:22:07.224204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224171 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224225 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224316 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224360 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224386 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224407 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b2p5\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224444 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224484 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224498 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224620 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224894 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224736 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.224894 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.224759 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.225162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.225136 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.225162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.225153 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.227182 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.227159 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.227506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.227486 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.280378 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.280322 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.280562 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.280544 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b2p5\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5\") pod \"e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.342498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.342474 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:07.592300 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:22:07.592204 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:22:07.775845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.775819 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw"] Nov 01 06:22:07.776570 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:22:07.776547 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cd4519_8810_4b9a_bc09_08214020d1f3.slice/crio-ee40c0fd12b254eaab8f83ddbd959035d51cadcf80a79c49464072f5b843b6fb WatchSource:0}: Error finding container ee40c0fd12b254eaab8f83ddbd959035d51cadcf80a79c49464072f5b843b6fb: Status 404 returned error can't find the container with id ee40c0fd12b254eaab8f83ddbd959035d51cadcf80a79c49464072f5b843b6fb Nov 01 06:22:07.778703 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.778671 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:22:07.778794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.778744 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:22:07.778794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.778786 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:22:07.792063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:07.792040 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" event={"ID":"53cd4519-8810-4b9a-bc09-08214020d1f3","Type":"ContainerStarted","Data":"ee40c0fd12b254eaab8f83ddbd959035d51cadcf80a79c49464072f5b843b6fb"} Nov 01 06:22:08.798313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:08.798262 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" event={"ID":"53cd4519-8810-4b9a-bc09-08214020d1f3","Type":"ContainerStarted","Data":"da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603"} Nov 01 06:22:09.343180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:09.343151 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:09.348136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:09.348115 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:09.464507 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:09.464467 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podStartSLOduration=3.464455257 podStartE2EDuration="3.464455257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:09.011466575 +0000 UTC m=+744.564255766" watchObservedRunningTime="2026-11-01 06:22:09.464455257 +0000 UTC m=+745.017244448" Nov 01 06:22:09.804098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:09.804070 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:09.805110 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:09.805088 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:15.430716 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.430678 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h"] Nov 01 06:22:15.435729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.435710 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:15.540658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.540613 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ef9b55\"" Nov 01 06:22:15.561318 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.561263 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h"] Nov 01 06:22:15.602091 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.601938 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:15.602091 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.602017 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwss\" (UniqueName: \"kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss\") pod \"payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:15.703299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.703195 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:15.703569 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.703536 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwss\" (UniqueName: \"kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss\") pod \"payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:15.704002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.703984 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:15.760399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:15.760376 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwss\" (UniqueName: \"kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss\") pod \"payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:16.049589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:16.049515 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:16.417267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:16.417243 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h"] Nov 01 06:22:16.418878 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:22:16.418850 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3583442f_cf21_4bd8_884d_ec9ef8fff3c2.slice/crio-e7ad85795cf24854402c71bab5432e15a70e79888c05dd00c85c5d702de62baa WatchSource:0}: Error finding container e7ad85795cf24854402c71bab5432e15a70e79888c05dd00c85c5d702de62baa: Status 404 returned error can't find the container with id e7ad85795cf24854402c71bab5432e15a70e79888c05dd00c85c5d702de62baa Nov 01 06:22:16.837926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:16.837848 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" event={"ID":"3583442f-cf21-4bd8-884d-ec9ef8fff3c2","Type":"ContainerStarted","Data":"7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514"} Nov 01 06:22:16.837926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:16.837889 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" event={"ID":"3583442f-cf21-4bd8-884d-ec9ef8fff3c2","Type":"ContainerStarted","Data":"e7ad85795cf24854402c71bab5432e15a70e79888c05dd00c85c5d702de62baa"} Nov 01 06:22:16.838405 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:16.837969 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:16.951139 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:16.951080 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" podStartSLOduration=1.951065846 podStartE2EDuration="1.951065846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:16.950964881 +0000 UTC m=+752.503754074" watchObservedRunningTime="2026-11-01 06:22:16.951065846 +0000 UTC m=+752.503855037" Nov 01 06:22:27.845736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:27.845659 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:22:35.551313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.551265 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx"] Nov 01 06:22:35.554781 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.554767 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.602599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.602576 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.602694 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.602607 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjll\" (UniqueName: \"kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.602694 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.602628 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.602758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.602700 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.606533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.606508 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ef9b55-serving-cert\"" Nov 01 06:22:35.658243 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.658218 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx"] Nov 01 06:22:35.703421 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.703388 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.703554 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.703525 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.704010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.703751 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjll\" (UniqueName: \"kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.704010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.703809 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.704010 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:22:35.703812 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-serving-cert: secret "maas-api-e2e-inf-b-ef9b55-serving-cert" not found Nov 01 06:22:35.704010 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:22:35.703915 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls podName:caf97c07-4c8d-49c2-9d6c-45d6306d9d40 nodeName:}" failed. No retries permitted until 2026-11-01 06:22:36.203895127 +0000 UTC m=+771.756684304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls") pod "maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" (UID: "caf97c07-4c8d-49c2-9d6c-45d6306d9d40") : secret "maas-api-e2e-inf-b-ef9b55-serving-cert" not found Nov 01 06:22:35.704220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.704103 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.706218 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.706200 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:35.760114 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:35.760089 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjll\" (UniqueName: \"kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:36.210821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:36.210782 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:36.213366 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:36.213337 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:36.467107 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:36.467041 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:36.836718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:36.836685 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx"] Nov 01 06:22:36.838243 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:22:36.838206 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf97c07_4c8d_49c2_9d6c_45d6306d9d40.slice/crio-ba11405630661957daaf768d23105f731a8cab861f8c41261c7840177630f775 WatchSource:0}: Error finding container ba11405630661957daaf768d23105f731a8cab861f8c41261c7840177630f775: Status 404 returned error can't find the container with id ba11405630661957daaf768d23105f731a8cab861f8c41261c7840177630f775 Nov 01 06:22:36.936726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:36.936698 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" event={"ID":"caf97c07-4c8d-49c2-9d6c-45d6306d9d40","Type":"ContainerStarted","Data":"ba11405630661957daaf768d23105f731a8cab861f8c41261c7840177630f775"} Nov 01 06:22:37.591300 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:22:37.591243 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:22:37.950429 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:37.950399 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" event={"ID":"caf97c07-4c8d-49c2-9d6c-45d6306d9d40","Type":"ContainerStarted","Data":"54c87f43bc73c7f3723aa9ee0b7f83892bf62add2f5491848deb2ef37ccb87e4"} Nov 01 06:22:37.950857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:37.950525 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:38.111204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:38.111160 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" podStartSLOduration=2.125090175 podStartE2EDuration="3.111147329s" totalImagesPullingTime="986.057154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:22:38.108621986 +0000 UTC m=+773.661411223" watchObservedRunningTime="2026-11-01 06:22:38.111147329 +0000 UTC m=+773.663936520" Nov 01 06:22:43.960866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:43.960827 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:22:55.428762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:55.428726 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:57606->10.132.0.52:15021: read: connection reset by peer" start-of-body= Nov 01 06:22:55.429203 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:55.428792 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:57606->10.132.0.52:15021: read: connection reset by peer" Nov 01 06:22:56.037320 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:56.037268 4914 generic.go:356] "Generic (PLEG): container finished" podID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerID="da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603" exitCode=0 Nov 01 06:22:56.037487 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:56.037346 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" event={"ID":"53cd4519-8810-4b9a-bc09-08214020d1f3","Type":"ContainerDied","Data":"da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603"} Nov 01 06:22:56.037717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:56.037700 4914 scope.go:122] "RemoveContainer" containerID="da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603" Nov 01 06:22:56.038980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:56.038950 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:22:56.039099 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:56.039061 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:22:56.039159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:56.039109 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:22:57.047739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:57.047703 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" event={"ID":"53cd4519-8810-4b9a-bc09-08214020d1f3","Type":"ContainerStarted","Data":"dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673"} Nov 01 06:22:57.343087 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:57.343004 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:22:57.343226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:57.343148 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 06:22:57.343226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:57.343191 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 06:22:58.342986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:58.342953 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 06:22:58.343456 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:58.343015 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 06:22:59.343028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:59.342995 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 06:22:59.343409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:22:59.343048 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 06:23:00.343705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:00.343674 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 06:23:00.344070 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:00.343739 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 06:23:01.342830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:01.342797 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 06:23:01.342994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:01.342856 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 06:23:02.343646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:02.343609 4914 patch_prober.go:29] interesting pod/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 01 06:23:02.344221 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:02.344190 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 01 06:23:03.346726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:03.346693 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:23:03.347182 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:03.347073 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:23:03.347640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:03.347625 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:23:07.595725 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:23:07.595644 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:23:09.882314 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.882269 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb"] Nov 01 06:23:09.886104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.886087 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:09.902800 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.902779 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vbk\" (UniqueName: \"kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk\") pod \"payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:09.902909 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.902811 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:09.938131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.938113 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c79abe\"" Nov 01 06:23:09.993086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.993050 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb"] Nov 01 06:23:09.993479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:09.993459 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c79abe-dockercfg-q6jdl\"" Nov 01 06:23:10.025978 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:10.025941 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vbk\" (UniqueName: \"kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk\") pod \"payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:10.026690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:10.026662 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:10.027608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:10.027572 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:10.085240 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:10.085216 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vbk\" (UniqueName: \"kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk\") pod \"payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:10.199405 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:10.199351 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:10.572393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:10.572368 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb"] Nov 01 06:23:10.574443 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:23:10.574415 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7e977e_6407_40b1_9d9a_656750d3e96a.slice/crio-cd1bbd3071cb7eae9d42d413f6bd86653bff2f7dbd75cbbd13627bddd784eb62 WatchSource:0}: Error finding container cd1bbd3071cb7eae9d42d413f6bd86653bff2f7dbd75cbbd13627bddd784eb62: Status 404 returned error can't find the container with id cd1bbd3071cb7eae9d42d413f6bd86653bff2f7dbd75cbbd13627bddd784eb62 Nov 01 06:23:11.136484 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:11.136458 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" event={"ID":"0a7e977e-6407-40b1-9d9a-656750d3e96a","Type":"ContainerStarted","Data":"cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9"} Nov 01 06:23:11.136873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:11.136514 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" event={"ID":"0a7e977e-6407-40b1-9d9a-656750d3e96a","Type":"ContainerStarted","Data":"cd1bbd3071cb7eae9d42d413f6bd86653bff2f7dbd75cbbd13627bddd784eb62"} Nov 01 06:23:11.136873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:11.136654 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:11.248888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:11.248826 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" podStartSLOduration=2.248805966 podStartE2EDuration="2.248805966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:11.247939215 +0000 UTC m=+806.800728403" watchObservedRunningTime="2026-11-01 06:23:11.248805966 +0000 UTC m=+806.801595211" Nov 01 06:23:22.145084 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:22.145054 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 06:23:26.198220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.198188 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx"] Nov 01 06:23:26.202168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.202153 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.257255 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.257234 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-190724-openshift-default-dockercfg-vr7pw\"" Nov 01 06:23:26.293264 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293237 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dsx\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293296 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293382 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293475 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293423 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293535 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293579 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293705 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293751 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.293812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.293799 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.305383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.305362 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx"] Nov 01 06:23:26.394907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.394886 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.394918 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.394937 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395063 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dsx\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395115 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395183 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395163 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395194 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395248 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395295 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395377 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.395991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.395966 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.396085 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.396033 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.396210 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.396181 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.396307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.396258 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.398632 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.398607 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.399986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.399951 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.451695 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.451643 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.451950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.451928 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dsx\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.514786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.514763 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:26.932594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.932570 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx"] Nov 01 06:23:26.933827 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:23:26.933801 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371f0896_9ad1_440c_b439_4ed59b3144bf.slice/crio-5d59e88f8df034c40021c35e103aa51609b707d87745fb2124167fc81508f2ee WatchSource:0}: Error finding container 5d59e88f8df034c40021c35e103aa51609b707d87745fb2124167fc81508f2ee: Status 404 returned error can't find the container with id 5d59e88f8df034c40021c35e103aa51609b707d87745fb2124167fc81508f2ee Nov 01 06:23:26.936031 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.936001 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:23:26.936096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.936061 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:23:26.936096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:26.936086 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:23:27.217527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:27.217452 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" event={"ID":"371f0896-9ad1-440c-b439-4ed59b3144bf","Type":"ContainerStarted","Data":"7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883"} Nov 01 06:23:27.217527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:27.217501 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" event={"ID":"371f0896-9ad1-440c-b439-4ed59b3144bf","Type":"ContainerStarted","Data":"5d59e88f8df034c40021c35e103aa51609b707d87745fb2124167fc81508f2ee"} Nov 01 06:23:27.333347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:27.333104 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podStartSLOduration=1.333087089 podStartE2EDuration="1.333087089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:27.332247184 +0000 UTC m=+822.885036375" watchObservedRunningTime="2026-11-01 06:23:27.333087089 +0000 UTC m=+822.885876282" Nov 01 06:23:27.515935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:27.515810 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:27.517999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:27.517953 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:23:27.518288 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:27.518040 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 06:23:28.515731 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:28.515699 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:23:28.516077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:28.515752 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 06:23:29.516054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:29.516017 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:23:29.516420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:29.516080 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 06:23:30.515644 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:30.515606 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:23:30.515826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:30.515664 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 06:23:31.515527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:31.515497 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:23:31.515882 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:31.515546 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 06:23:32.515418 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:32.515383 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 01 06:23:32.515611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:32.515436 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 01 06:23:33.519514 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:33.519487 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:33.519896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:33.519879 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:33.520553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:33.520532 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:23:37.590950 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:23:37.590917 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:23:37.850032 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:37.849995 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg"] Nov 01 06:23:37.853902 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:37.853886 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:37.957213 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:37.957180 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c79abe-serving-cert\"" Nov 01 06:23:37.959396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:37.959375 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg"] Nov 01 06:23:38.031640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.031614 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.031774 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.031730 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.031774 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.031764 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.031883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.031782 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lggb\" (UniqueName: \"kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.133067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.132975 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.133373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.133348 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.133481 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.133398 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.133481 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.133422 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lggb\" (UniqueName: \"kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.133885 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:23:38.133860 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-serving-cert: secret "maas-api-e2e-extmod-a-c79abe-serving-cert" not found Nov 01 06:23:38.134006 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:23:38.133920 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls podName:393cafe2-a612-4725-b031-fd45c6496b80 nodeName:}" failed. No retries permitted until 2026-11-01 06:23:38.633906132 +0000 UTC m=+834.186695315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls") pod "maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" (UID: "393cafe2-a612-4725-b031-fd45c6496b80") : secret "maas-api-e2e-extmod-a-c79abe-serving-cert" not found Nov 01 06:23:38.134674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.134650 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.137364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.137341 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.191899 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.191875 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lggb\" (UniqueName: \"kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.637951 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.637922 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.640208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.640189 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:38.768952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:38.768924 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:39.148805 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:39.148777 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg"] Nov 01 06:23:39.150624 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:23:39.150596 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393cafe2_a612_4725_b031_fd45c6496b80.slice/crio-8d24535fd0ee16acf9ad1accd27327ca8c54529115e588d941f032ccb8ba4829 WatchSource:0}: Error finding container 8d24535fd0ee16acf9ad1accd27327ca8c54529115e588d941f032ccb8ba4829: Status 404 returned error can't find the container with id 8d24535fd0ee16acf9ad1accd27327ca8c54529115e588d941f032ccb8ba4829 Nov 01 06:23:39.311668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:39.311630 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" event={"ID":"393cafe2-a612-4725-b031-fd45c6496b80","Type":"ContainerStarted","Data":"8d24535fd0ee16acf9ad1accd27327ca8c54529115e588d941f032ccb8ba4829"} Nov 01 06:23:40.317762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:40.317673 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" event={"ID":"393cafe2-a612-4725-b031-fd45c6496b80","Type":"ContainerStarted","Data":"cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69"} Nov 01 06:23:40.317762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:40.317760 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:23:40.431896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:40.431852 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" podStartSLOduration=2.5735774129999998 podStartE2EDuration="3.431838187s" totalImagesPullingTime="858.260774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:23:40.429596283 +0000 UTC m=+835.982385473" watchObservedRunningTime="2026-11-01 06:23:40.431838187 +0000 UTC m=+835.984627378" Nov 01 06:23:46.327847 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:23:46.327817 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 06:24:07.597911 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:24:07.591327 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:24:27.458612 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.458580 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r"] Nov 01 06:24:27.463050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.463031 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.518758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.518732 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r"] Nov 01 06:24:27.565141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.565120 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:24:27.623090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.623054 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf"] Nov 01 06:24:27.628125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.628098 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.636432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.636399 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztv9\" (UniqueName: \"kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9\") pod \"payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.636884 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.636838 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.681529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.681496 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-dc6e48-dockercfg-cgzrm\"" Nov 01 06:24:27.684107 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.684081 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf"] Nov 01 06:24:27.737970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.737897 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.737970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.737932 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chff8\" (UniqueName: \"kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8\") pod \"payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.738136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.737976 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.738136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.738122 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztv9\" (UniqueName: \"kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9\") pod \"payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.738752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.738734 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.794747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.794725 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztv9\" (UniqueName: \"kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9\") pod \"payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:27.838833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.838809 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chff8\" (UniqueName: \"kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8\") pod \"payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.838936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.838850 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.839557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.839535 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.894867 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.894843 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chff8\" (UniqueName: \"kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8\") pod \"payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:27.964836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:27.964818 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:28.078224 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.078194 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:28.351160 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.351138 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf"] Nov 01 06:24:28.352712 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:24:28.352689 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c047da_a881_4066_ad57_73e37e66980c.slice/crio-467cf3481ac279021cea000ebe1a10b6f40ffb3ba2e71340fd046603b95980d2 WatchSource:0}: Error finding container 467cf3481ac279021cea000ebe1a10b6f40ffb3ba2e71340fd046603b95980d2: Status 404 returned error can't find the container with id 467cf3481ac279021cea000ebe1a10b6f40ffb3ba2e71340fd046603b95980d2 Nov 01 06:24:28.465786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.465764 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r"] Nov 01 06:24:28.467174 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:24:28.467148 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284136a9_8bc0_4a17_8c61_a0bbc1b87f90.slice/crio-020eeb118e8da2867ec57a36abb01b8933def83991edc570888198dc9367ac74 WatchSource:0}: Error finding container 020eeb118e8da2867ec57a36abb01b8933def83991edc570888198dc9367ac74: Status 404 returned error can't find the container with id 020eeb118e8da2867ec57a36abb01b8933def83991edc570888198dc9367ac74 Nov 01 06:24:28.576773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.576744 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" event={"ID":"22c047da-a881-4066-ad57-73e37e66980c","Type":"ContainerStarted","Data":"acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07"} Nov 01 06:24:28.576913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.576797 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" event={"ID":"22c047da-a881-4066-ad57-73e37e66980c","Type":"ContainerStarted","Data":"467cf3481ac279021cea000ebe1a10b6f40ffb3ba2e71340fd046603b95980d2"} Nov 01 06:24:28.576975 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.576936 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:28.578368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.578347 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" event={"ID":"284136a9-8bc0-4a17-8c61-a0bbc1b87f90","Type":"ContainerStarted","Data":"916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456"} Nov 01 06:24:28.578470 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.578389 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" event={"ID":"284136a9-8bc0-4a17-8c61-a0bbc1b87f90","Type":"ContainerStarted","Data":"020eeb118e8da2867ec57a36abb01b8933def83991edc570888198dc9367ac74"} Nov 01 06:24:28.578470 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.578419 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:28.693116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.693040 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" podStartSLOduration=1.6930294190000001 podStartE2EDuration="1.693029419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:28.69087524 +0000 UTC m=+884.243664443" watchObservedRunningTime="2026-11-01 06:24:28.693029419 +0000 UTC m=+884.245818670" Nov 01 06:24:28.821098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:28.814770 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" podStartSLOduration=1.81475061 podStartE2EDuration="1.81475061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:24:28.814006863 +0000 UTC m=+884.366796058" watchObservedRunningTime="2026-11-01 06:24:28.81475061 +0000 UTC m=+884.367539801" Nov 01 06:24:37.592068 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:24:37.592036 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:24:39.585859 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:39.585828 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 06:24:39.586333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:39.585894 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 06:24:44.701498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:44.701469 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:24:44.703927 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:44.703907 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:24:58.965520 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:58.965489 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk"] Nov 01 06:24:58.970540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:58.970522 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.017352 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.017317 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.017452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.017382 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.017520 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.017499 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnbz\" (UniqueName: \"kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.017598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.017529 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.022005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.021984 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-dc6e48-serving-cert\"" Nov 01 06:24:59.074120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.074097 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk"] Nov 01 06:24:59.118659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.118629 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnbz\" (UniqueName: \"kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.118801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.118677 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.118801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.118742 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.118801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.118794 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.119059 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:24:59.119039 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-serving-cert: secret "maas-api-e2e-extmod-b-dc6e48-serving-cert" not found Nov 01 06:24:59.119123 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:24:59.119103 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls podName:2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a nodeName:}" failed. No retries permitted until 2026-11-01 06:24:59.619084025 +0000 UTC m=+915.171873209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls") pod "maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" (UID: "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a") : secret "maas-api-e2e-extmod-b-dc6e48-serving-cert" not found Nov 01 06:24:59.119543 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.119521 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.121561 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.121527 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.177676 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.177650 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnbz\" (UniqueName: \"kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.634665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.634630 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.641170 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.641147 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:24:59.883052 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:24:59.883021 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:25:00.264506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:00.264353 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk"] Nov 01 06:25:00.270329 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:25:00.270248 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd95b96_99e7_45f6_b4e1_38f6ef9eb63a.slice/crio-64898fdf3f9e2f9e8a3fbc46642d136128be9e8f8bdadf6cfce9167190603151 WatchSource:0}: Error finding container 64898fdf3f9e2f9e8a3fbc46642d136128be9e8f8bdadf6cfce9167190603151: Status 404 returned error can't find the container with id 64898fdf3f9e2f9e8a3fbc46642d136128be9e8f8bdadf6cfce9167190603151 Nov 01 06:25:00.746150 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:00.746118 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" event={"ID":"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a","Type":"ContainerStarted","Data":"64898fdf3f9e2f9e8a3fbc46642d136128be9e8f8bdadf6cfce9167190603151"} Nov 01 06:25:01.753061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:01.753029 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" event={"ID":"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a","Type":"ContainerStarted","Data":"0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2"} Nov 01 06:25:01.753518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:01.753091 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:25:01.866964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:01.866921 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" podStartSLOduration=2.924362244 podStartE2EDuration="3.866908873s" totalImagesPullingTime="942.546629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:01.865407263 +0000 UTC m=+917.418196457" watchObservedRunningTime="2026-11-01 06:25:01.866908873 +0000 UTC m=+917.419698061" Nov 01 06:25:07.591937 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:25:07.591901 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:25:07.763113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:07.763086 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 06:25:37.596072 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:25:37.596035 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:25:53.383258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.378746 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn"] Nov 01 06:25:53.384121 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.384092 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.428625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.428594 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn"] Nov 01 06:25:53.450291 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.450247 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\"" Nov 01 06:25:53.450599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.450580 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\"" Nov 01 06:25:53.471838 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.471806 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\"" Nov 01 06:25:53.472144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.472125 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:25:53.472414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.472386 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\"" Nov 01 06:25:53.501117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.501086 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\"" Nov 01 06:25:53.519327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.519286 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n48v\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.519831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.519802 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.520098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.520077 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.520353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.520331 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.621768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.621737 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.621908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.621811 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.621908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.621848 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.621982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.621927 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n48v\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.622564 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.622499 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.622564 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.622586 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.624210 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.624189 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.681175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.681127 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n48v\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:53.728122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:53.728097 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:54.218382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:54.212616 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn"] Nov 01 06:25:54.218382 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:25:54.213944 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95db6e8_52b7_485f_969f_42cc74d6034f.slice/crio-9ea97fee1ebffbb9e0694431025135160e67c86c14392fa87adf29dc80a45bba WatchSource:0}: Error finding container 9ea97fee1ebffbb9e0694431025135160e67c86c14392fa87adf29dc80a45bba: Status 404 returned error can't find the container with id 9ea97fee1ebffbb9e0694431025135160e67c86c14392fa87adf29dc80a45bba Nov 01 06:25:55.046069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:55.046021 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" event={"ID":"d95db6e8-52b7-485f-969f-42cc74d6034f","Type":"ContainerStarted","Data":"418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746"} Nov 01 06:25:55.046465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:55.046094 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" event={"ID":"d95db6e8-52b7-485f-969f-42cc74d6034f","Type":"ContainerStarted","Data":"9ea97fee1ebffbb9e0694431025135160e67c86c14392fa87adf29dc80a45bba"} Nov 01 06:25:55.046465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:55.046142 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:25:55.159408 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:55.159358 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" podStartSLOduration=2.159339155 podStartE2EDuration="2.159339155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:55.158733049 +0000 UTC m=+970.711522241" watchObservedRunningTime="2026-11-01 06:25:55.159339155 +0000 UTC m=+970.712128346" Nov 01 06:25:57.401315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.400560 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2"] Nov 01 06:25:57.416340 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.410996 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.477390 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.477355 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2"] Nov 01 06:25:57.570314 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.570252 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4pg9\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.574009 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.573976 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.574344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.574323 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.574572 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.574552 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.574848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.574826 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.575109 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.575063 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.575375 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.575326 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.575625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.575572 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.575825 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.575808 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.673051 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.672974 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq"] Nov 01 06:25:57.676736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.676722 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.677008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.676980 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677081 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677059 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4pg9\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677320 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677259 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677422 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677325 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677422 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677375 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677532 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677459 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677597 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677576 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.677653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677634 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.678592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677842 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.678592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677927 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.678592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.677965 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.678592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.678030 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.678592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.678333 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.678592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.678342 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.679604 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.679582 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.680386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.680365 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.730386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.730365 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd9d24\"" Nov 01 06:25:57.730386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.730380 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd9d24-dockercfg-8f4rq\"" Nov 01 06:25:57.733787 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.733763 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq"] Nov 01 06:25:57.735226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.735182 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.736676 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.736641 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4pg9\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.743383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.743160 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:57.796305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.781965 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.796305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.782026 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8ts\" (UniqueName: \"kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.840871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.840831 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw"] Nov 01 06:25:57.851712 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.851676 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:57.883979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.883938 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.884258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.884230 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8ts\" (UniqueName: \"kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.884541 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.884507 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:57.884719 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.884702 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5557g\" (UniqueName: \"kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g\") pod \"payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:57.886418 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.886004 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.896300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.894695 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw"] Nov 01 06:25:57.943898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.943830 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8ts\" (UniqueName: \"kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts\") pod \"payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:57.986141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.986114 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:57.986255 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.986147 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5557g\" (UniqueName: \"kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g\") pod \"payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:57.986749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:57.986730 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:58.000936 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.000915 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:58.043389 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.043366 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5557g\" (UniqueName: \"kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g\") pod \"payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:58.169523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.169498 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:58.260988 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:25:58.260956 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d707cb0_5872_4d7d_893b_49b553872b0c.slice/crio-7aba113949314ad26d695b7de560ab0e95ddbbc79d1629a0a3c0abcc168be3d8 WatchSource:0}: Error finding container 7aba113949314ad26d695b7de560ab0e95ddbbc79d1629a0a3c0abcc168be3d8: Status 404 returned error can't find the container with id 7aba113949314ad26d695b7de560ab0e95ddbbc79d1629a0a3c0abcc168be3d8 Nov 01 06:25:58.261177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.261034 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2"] Nov 01 06:25:58.263319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.263269 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:25:58.263384 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.263351 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:25:58.263431 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.263382 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:25:58.375925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.375889 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq"] Nov 01 06:25:58.378846 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:25:58.378813 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20434eee_bec4_4907_a08b_0733d04b63d6.slice/crio-0d2ffac248ef6375a52a6ab63c39f7c305bf5bc3a09002ff3ecbf9c14d6f213b WatchSource:0}: Error finding container 0d2ffac248ef6375a52a6ab63c39f7c305bf5bc3a09002ff3ecbf9c14d6f213b: Status 404 returned error can't find the container with id 0d2ffac248ef6375a52a6ab63c39f7c305bf5bc3a09002ff3ecbf9c14d6f213b Nov 01 06:25:58.543228 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:25:58.543204 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91c8651_5499_45dd_af96_f04855dafb5a.slice/crio-6c784cf2c2d7b07859ccffe0f9f411b41339cc4f6a9901e6aa3f8aaec2e4a350 WatchSource:0}: Error finding container 6c784cf2c2d7b07859ccffe0f9f411b41339cc4f6a9901e6aa3f8aaec2e4a350: Status 404 returned error can't find the container with id 6c784cf2c2d7b07859ccffe0f9f411b41339cc4f6a9901e6aa3f8aaec2e4a350 Nov 01 06:25:58.543550 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:58.543394 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw"] Nov 01 06:25:59.077839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.077795 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" event={"ID":"5d707cb0-5872-4d7d-893b-49b553872b0c","Type":"ContainerStarted","Data":"ab477cf78ef3da68601c4fe9564a4bc8ee3f38216bc9850ba3418ed010743269"} Nov 01 06:25:59.078045 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.077858 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" event={"ID":"5d707cb0-5872-4d7d-893b-49b553872b0c","Type":"ContainerStarted","Data":"7aba113949314ad26d695b7de560ab0e95ddbbc79d1629a0a3c0abcc168be3d8"} Nov 01 06:25:59.079495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.079464 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" event={"ID":"20434eee-bec4-4907-a08b-0733d04b63d6","Type":"ContainerStarted","Data":"5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd"} Nov 01 06:25:59.079627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.079513 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" event={"ID":"20434eee-bec4-4907-a08b-0733d04b63d6","Type":"ContainerStarted","Data":"0d2ffac248ef6375a52a6ab63c39f7c305bf5bc3a09002ff3ecbf9c14d6f213b"} Nov 01 06:25:59.079627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.079540 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:25:59.083858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.083833 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" event={"ID":"a91c8651-5499-45dd-af96-f04855dafb5a","Type":"ContainerStarted","Data":"0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229"} Nov 01 06:25:59.083958 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.083879 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" event={"ID":"a91c8651-5499-45dd-af96-f04855dafb5a","Type":"ContainerStarted","Data":"6c784cf2c2d7b07859ccffe0f9f411b41339cc4f6a9901e6aa3f8aaec2e4a350"} Nov 01 06:25:59.083958 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.083936 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:25:59.194986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.194939 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podStartSLOduration=2.194926182 podStartE2EDuration="2.194926182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:59.191782868 +0000 UTC m=+974.744572059" watchObservedRunningTime="2026-11-01 06:25:59.194926182 +0000 UTC m=+974.747715373" Nov 01 06:25:59.305907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.305848 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" podStartSLOduration=2.305832126 podStartE2EDuration="2.305832126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:59.303631738 +0000 UTC m=+974.856420933" watchObservedRunningTime="2026-11-01 06:25:59.305832126 +0000 UTC m=+974.858621318" Nov 01 06:25:59.744640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.744610 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:25:59.766415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.766384 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 01 06:25:59.766658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:25:59.766629 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 01 06:26:00.744131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:00.744101 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 01 06:26:00.744316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:00.744156 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 01 06:26:01.744218 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:01.744184 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 01 06:26:01.744610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:01.744234 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 01 06:26:02.744114 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:02.744078 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 01 06:26:02.744302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:02.744135 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 01 06:26:03.744748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:03.744715 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 01 06:26:03.745144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:03.744769 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 01 06:26:04.744092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:04.744052 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 01 06:26:04.744311 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:04.744095 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 01 06:26:05.748656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:05.748627 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:26:05.749167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:05.749145 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:26:05.749715 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:05.749698 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:26:05.914874 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:05.914821 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" podStartSLOduration=8.914808464 podStartE2EDuration="8.914808464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:25:59.434179186 +0000 UTC m=+974.986968377" watchObservedRunningTime="2026-11-01 06:26:05.914808464 +0000 UTC m=+981.467597713" Nov 01 06:26:06.057107 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:06.057034 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:26:07.592020 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:26:07.591988 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:26:10.092434 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:10.092409 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:26:10.092806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:10.092465 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:26:25.575193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.575160 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4"] Nov 01 06:26:25.580202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.580187 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.635345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.635317 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4"] Nov 01 06:26:25.681175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.681146 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bd9d24-serving-cert\"" Nov 01 06:26:25.710401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.710371 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6gn\" (UniqueName: \"kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.710401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.710402 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.710578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.710478 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.710578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.710535 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.811573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.811545 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6gn\" (UniqueName: \"kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.812354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.812328 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.812563 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.812544 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.812865 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.812846 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.813348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.813073 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.813348 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:26:25.812734 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-serving-cert: secret "maas-api-e2e-ipp-a-bd9d24-serving-cert" not found Nov 01 06:26:25.813348 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:26:25.813340 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls podName:826ee346-5b8e-45af-bdc4-15d2d09ff953 nodeName:}" failed. No retries permitted until 2026-11-01 06:26:26.313321224 +0000 UTC m=+1001.866110406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls") pod "maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" (UID: "826ee346-5b8e-45af-bdc4-15d2d09ff953") : secret "maas-api-e2e-ipp-a-bd9d24-serving-cert" not found Nov 01 06:26:25.832352 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.832142 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:25.877561 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:25.875726 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6gn\" (UniqueName: \"kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:26.319711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:26.319675 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:26.322220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:26.322197 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:26.493123 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:26.493096 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:26.866187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:26.866161 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4"] Nov 01 06:26:26.867873 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:26:26.867838 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826ee346_5b8e_45af_bdc4_15d2d09ff953.slice/crio-3df483b8da3192828020d0b1762a5abf1abeec385c63b025242683538060be91 WatchSource:0}: Error finding container 3df483b8da3192828020d0b1762a5abf1abeec385c63b025242683538060be91: Status 404 returned error can't find the container with id 3df483b8da3192828020d0b1762a5abf1abeec385c63b025242683538060be91 Nov 01 06:26:26.869257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:26.869239 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:26:27.264547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:27.264478 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" event={"ID":"826ee346-5b8e-45af-bdc4-15d2d09ff953","Type":"ContainerStarted","Data":"3df483b8da3192828020d0b1762a5abf1abeec385c63b025242683538060be91"} Nov 01 06:26:28.294744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:28.294675 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" event={"ID":"826ee346-5b8e-45af-bdc4-15d2d09ff953","Type":"ContainerStarted","Data":"d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81"} Nov 01 06:26:28.299677 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:28.296374 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:28.411768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:28.411719 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" podStartSLOduration=2.5088348160000002 podStartE2EDuration="3.411703201s" totalImagesPullingTime="902.868385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:28.410947355 +0000 UTC m=+1003.963736547" watchObservedRunningTime="2026-11-01 06:26:28.411703201 +0000 UTC m=+1003.964492442" Nov 01 06:26:35.348945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:35.348897 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:26:37.095840 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.095807 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8"] Nov 01 06:26:37.099693 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.099676 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.144344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.144321 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.144454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.144363 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rxt\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.144536 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.144519 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.144579 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.144546 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.152542 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.152520 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\"" Nov 01 06:26:37.204439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.204407 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8"] Nov 01 06:26:37.246054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.246027 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.246144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.246062 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rxt\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.246178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.246159 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.246232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.246178 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.246732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.246711 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.246958 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.246819 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.248580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.248561 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.302576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.302551 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rxt\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.411938 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.411878 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:37.591956 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:26:37.591918 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:26:37.783998 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:37.783965 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8"] Nov 01 06:26:38.371127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:38.370799 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" event={"ID":"06843d49-e37d-435f-9c33-1d3717789f51","Type":"ContainerStarted","Data":"3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de"} Nov 01 06:26:38.371127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:38.370859 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" event={"ID":"06843d49-e37d-435f-9c33-1d3717789f51","Type":"ContainerStarted","Data":"a7d26aeb8aa821ba81723370f287cf65951ed0cac359ad650859905e549ec98d"} Nov 01 06:26:38.371127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:38.370944 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:38.484806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:38.484757 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" podStartSLOduration=1.484744485 podStartE2EDuration="1.484744485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:38.482891373 +0000 UTC m=+1014.035680567" watchObservedRunningTime="2026-11-01 06:26:38.484744485 +0000 UTC m=+1014.037533676" Nov 01 06:26:42.761053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.761022 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l"] Nov 01 06:26:42.765175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.765145 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.803923 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.803898 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.803945 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4x9\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.803962 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.804069 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.804096 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.804115 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.804135 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804268 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.804225 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.804342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.804330 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.875120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.875094 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l"] Nov 01 06:26:42.905406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.905375 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.905553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.905458 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.905553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.905528 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4x9\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.905672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.905554 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.905672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.905588 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.906206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.906177 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.906415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.906394 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.906544 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.906527 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.906686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.906669 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.908106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.906600 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.908106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.906716 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.908106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.907490 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.908750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.908716 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.909745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.909724 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.912969 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.910921 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.919638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.919613 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.963190 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.963169 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:42.963534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:42.963512 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4x9\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:43.079050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:43.078992 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:43.503122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:43.503098 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l"] Nov 01 06:26:43.504097 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:26:43.504072 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b4bc82_eb35_4d86_bba8_48e662e71f09.slice/crio-87e3e16a648f6628a9fead08b51762fa89e1df48812302016eb5b8c45af0b126 WatchSource:0}: Error finding container 87e3e16a648f6628a9fead08b51762fa89e1df48812302016eb5b8c45af0b126: Status 404 returned error can't find the container with id 87e3e16a648f6628a9fead08b51762fa89e1df48812302016eb5b8c45af0b126 Nov 01 06:26:43.506217 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:43.506188 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:26:43.506296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:43.506251 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:26:43.506339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:43.506301 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:26:44.417325 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:44.417266 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" event={"ID":"19b4bc82-eb35-4d86-bba8-48e662e71f09","Type":"ContainerStarted","Data":"7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad"} Nov 01 06:26:44.417713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:44.417346 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" event={"ID":"19b4bc82-eb35-4d86-bba8-48e662e71f09","Type":"ContainerStarted","Data":"87e3e16a648f6628a9fead08b51762fa89e1df48812302016eb5b8c45af0b126"} Nov 01 06:26:44.636006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:44.635961 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podStartSLOduration=2.635948533 podStartE2EDuration="2.635948533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:26:44.58451817 +0000 UTC m=+1020.137307379" watchObservedRunningTime="2026-11-01 06:26:44.635948533 +0000 UTC m=+1020.188737724" Nov 01 06:26:45.079248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:45.079216 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:45.080931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:45.080881 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 06:26:45.081049 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:45.080961 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 06:26:46.079960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:46.079920 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 06:26:46.080450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:46.079987 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 06:26:47.070919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:47.070881 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd"] Nov 01 06:26:47.071367 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:47.071327 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" containerID="cri-o://60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf" gracePeriod=30 Nov 01 06:26:47.080069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:47.080041 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 06:26:47.080440 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:47.080092 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 06:26:48.081621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:48.081585 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 06:26:48.082119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:48.081681 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 06:26:49.079919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:49.079882 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 06:26:49.080163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:49.079947 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 06:26:49.380490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:49.380459 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:26:49.561345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:49.560589 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn"] Nov 01 06:26:50.079833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:50.079799 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 01 06:26:50.079993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:50.079848 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 01 06:26:51.696102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:51.696067 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:51.696506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:51.696489 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:51.706075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:51.706052 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:26:52.325472 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.325450 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:26:52.473235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473201 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473268 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473316 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473380 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473422 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473448 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473527 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473519 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.473732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473627 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473731 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy\") pod \"38fa915f-666d-45ac-9b5f-839e54ac78eb\" (UID: \"38fa915f-666d-45ac-9b5f-839e54ac78eb\") " Nov 01 06:26:52.473871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473827 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.473871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.473858 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.474104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.474065 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.474254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.474238 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.474337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.474261 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.474337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.474310 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.474337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.474325 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.475301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.475248 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:52.476469 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.476436 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:26:52.476686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.476660 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:52.476752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.476705 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:26:52.477460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.477439 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx" pod "38fa915f-666d-45ac-9b5f-839e54ac78eb" (UID: "38fa915f-666d-45ac-9b5f-839e54ac78eb"). InnerVolumeSpecName "kube-api-access-vk2mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:52.484573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.484550 4914 generic.go:356] "Generic (PLEG): container finished" podID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerID="60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf" exitCode=0 Nov 01 06:26:52.484663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.484621 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" Nov 01 06:26:52.484733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.484674 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" event={"ID":"38fa915f-666d-45ac-9b5f-839e54ac78eb","Type":"ContainerDied","Data":"60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf"} Nov 01 06:26:52.484783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.484721 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd" event={"ID":"38fa915f-666d-45ac-9b5f-839e54ac78eb","Type":"ContainerDied","Data":"00b9197178a195610fb89eb2e461d3a18a123e2f26ee1f929f729c1d7567616e"} Nov 01 06:26:52.484783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.484757 4914 scope.go:122] "RemoveContainer" containerID="60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf" Nov 01 06:26:52.515298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.515257 4914 scope.go:122] "RemoveContainer" containerID="60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf" Nov 01 06:26:52.515651 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:26:52.515630 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf\": container with ID starting with 60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf not found: ID does not exist" containerID="60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf" Nov 01 06:26:52.515743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.515659 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf"} err="failed to get container status \"60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf\": rpc error: code = NotFound desc = could not find container \"60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf\": container with ID starting with 60af5f219f26915fe03844e634ddcf0d2eda8ca6fd606dd6e6ce99354943d5bf not found: ID does not exist" Nov 01 06:26:52.575586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.575560 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fa915f-666d-45ac-9b5f-839e54ac78eb-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.575586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.575580 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2mx\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-kube-api-access-vk2mx\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.575586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.575592 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.575801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.575600 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.575801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.575609 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fa915f-666d-45ac-9b5f-839e54ac78eb-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:52.648837 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.648810 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd"] Nov 01 06:26:52.652880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:52.652856 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-wvfkd"] Nov 01 06:26:54.561505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.561458 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" podUID="d95db6e8-52b7-485f-969f-42cc74d6034f" containerName="payload-processing" containerID="cri-o://418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746" gracePeriod=25 Nov 01 06:26:54.634370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.634339 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" path="/var/lib/kubelet/pods/38fa915f-666d-45ac-9b5f-839e54ac78eb/volumes" Nov 01 06:26:54.816702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.816644 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:26:54.899728 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.899703 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials\") pod \"d95db6e8-52b7-485f-969f-42cc74d6034f\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " Nov 01 06:26:54.899859 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.899736 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v\") pod \"d95db6e8-52b7-485f-969f-42cc74d6034f\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " Nov 01 06:26:54.899859 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.899835 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume\") pod \"d95db6e8-52b7-485f-969f-42cc74d6034f\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " Nov 01 06:26:54.899970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.899942 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume\") pod \"d95db6e8-52b7-485f-969f-42cc74d6034f\" (UID: \"d95db6e8-52b7-485f-969f-42cc74d6034f\") " Nov 01 06:26:54.900227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.900201 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume" pod "d95db6e8-52b7-485f-969f-42cc74d6034f" (UID: "d95db6e8-52b7-485f-969f-42cc74d6034f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:54.900380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.900262 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume" pod "d95db6e8-52b7-485f-969f-42cc74d6034f" (UID: "d95db6e8-52b7-485f-969f-42cc74d6034f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:26:54.900380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.900368 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:54.900380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.900381 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95db6e8-52b7-485f-969f-42cc74d6034f-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:54.901904 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.901876 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials" pod "d95db6e8-52b7-485f-969f-42cc74d6034f" (UID: "d95db6e8-52b7-485f-969f-42cc74d6034f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:54.902022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:54.902005 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v" pod "d95db6e8-52b7-485f-969f-42cc74d6034f" (UID: "d95db6e8-52b7-485f-969f-42cc74d6034f"). InnerVolumeSpecName "kube-api-access-9n48v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:26:55.001901 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.001874 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:55.001901 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.001897 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n48v\" (UniqueName: \"kubernetes.io/projected/d95db6e8-52b7-485f-969f-42cc74d6034f-kube-api-access-9n48v\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:26:55.504569 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.504524 4914 generic.go:356] "Generic (PLEG): container finished" podID="d95db6e8-52b7-485f-969f-42cc74d6034f" containerID="418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746" exitCode=0 Nov 01 06:26:55.504737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.504593 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" event={"ID":"d95db6e8-52b7-485f-969f-42cc74d6034f","Type":"ContainerDied","Data":"418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746"} Nov 01 06:26:55.504737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.504636 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" Nov 01 06:26:55.504737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.504654 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn" event={"ID":"d95db6e8-52b7-485f-969f-42cc74d6034f","Type":"ContainerDied","Data":"9ea97fee1ebffbb9e0694431025135160e67c86c14392fa87adf29dc80a45bba"} Nov 01 06:26:55.504737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.504678 4914 scope.go:122] "RemoveContainer" containerID="418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746" Nov 01 06:26:55.524033 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.524011 4914 scope.go:122] "RemoveContainer" containerID="418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746" Nov 01 06:26:55.524289 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:26:55.524258 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746\": container with ID starting with 418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746 not found: ID does not exist" containerID="418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746" Nov 01 06:26:55.524333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.524297 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746"} err="failed to get container status \"418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746\": rpc error: code = NotFound desc = could not find container \"418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746\": container with ID starting with 418970ac090a1e1f1e871b6664242e08cef755c20025fe12fc7f0da137852746 not found: ID does not exist" Nov 01 06:26:55.687499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.687470 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn"] Nov 01 06:26:55.737102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:55.737075 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c8ndmn"] Nov 01 06:26:56.615764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:26:56.615698 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95db6e8-52b7-485f-969f-42cc74d6034f" path="/var/lib/kubelet/pods/d95db6e8-52b7-485f-969f-42cc74d6034f/volumes" Nov 01 06:27:07.591890 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:27:07.591854 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:27:12.144887 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:12.144845 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8"] Nov 01 06:27:17.145760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.145724 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" podUID="06843d49-e37d-435f-9c33-1d3717789f51" containerName="payload-processing" containerID="cri-o://3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de" gracePeriod=25 Nov 01 06:27:17.398745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.398692 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:27:17.446806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.446785 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt\") pod \"06843d49-e37d-435f-9c33-1d3717789f51\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " Nov 01 06:27:17.446909 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.446813 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials\") pod \"06843d49-e37d-435f-9c33-1d3717789f51\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " Nov 01 06:27:17.446909 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.446846 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume\") pod \"06843d49-e37d-435f-9c33-1d3717789f51\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " Nov 01 06:27:17.447021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.446904 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume\") pod \"06843d49-e37d-435f-9c33-1d3717789f51\" (UID: \"06843d49-e37d-435f-9c33-1d3717789f51\") " Nov 01 06:27:17.447250 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.447220 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume" pod "06843d49-e37d-435f-9c33-1d3717789f51" (UID: "06843d49-e37d-435f-9c33-1d3717789f51"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:17.447395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.447367 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume" pod "06843d49-e37d-435f-9c33-1d3717789f51" (UID: "06843d49-e37d-435f-9c33-1d3717789f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:17.447619 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.447438 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:17.448927 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.448888 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt" pod "06843d49-e37d-435f-9c33-1d3717789f51" (UID: "06843d49-e37d-435f-9c33-1d3717789f51"). InnerVolumeSpecName "kube-api-access-w5rxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:17.449007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.448988 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials" pod "06843d49-e37d-435f-9c33-1d3717789f51" (UID: "06843d49-e37d-435f-9c33-1d3717789f51"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:17.548631 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.548609 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06843d49-e37d-435f-9c33-1d3717789f51-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:17.548631 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.548629 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5rxt\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-kube-api-access-w5rxt\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:17.548771 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.548641 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06843d49-e37d-435f-9c33-1d3717789f51-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:17.662305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.662230 4914 generic.go:356] "Generic (PLEG): container finished" podID="06843d49-e37d-435f-9c33-1d3717789f51" containerID="3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de" exitCode=0 Nov 01 06:27:17.662305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.662294 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" Nov 01 06:27:17.662454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.662322 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" event={"ID":"06843d49-e37d-435f-9c33-1d3717789f51","Type":"ContainerDied","Data":"3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de"} Nov 01 06:27:17.662454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.662373 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8" event={"ID":"06843d49-e37d-435f-9c33-1d3717789f51","Type":"ContainerDied","Data":"a7d26aeb8aa821ba81723370f287cf65951ed0cac359ad650859905e549ec98d"} Nov 01 06:27:17.662454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.662396 4914 scope.go:122] "RemoveContainer" containerID="3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de" Nov 01 06:27:17.671608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.671593 4914 scope.go:122] "RemoveContainer" containerID="3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de" Nov 01 06:27:17.671836 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:27:17.671818 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de\": container with ID starting with 3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de not found: ID does not exist" containerID="3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de" Nov 01 06:27:17.671890 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.671844 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de"} err="failed to get container status \"3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de\": rpc error: code = NotFound desc = could not find container \"3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de\": container with ID starting with 3d65ab3903893c0d3cddab010a2684bc208c89af075f5e325a7c40801c05b2de not found: ID does not exist" Nov 01 06:27:17.826657 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.826630 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8"] Nov 01 06:27:17.832620 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:17.832598 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788cdrt8"] Nov 01 06:27:18.616867 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:18.616839 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06843d49-e37d-435f-9c33-1d3717789f51" path="/var/lib/kubelet/pods/06843d49-e37d-435f-9c33-1d3717789f51/volumes" Nov 01 06:27:19.947350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:19.947313 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw"] Nov 01 06:27:20.061862 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.061827 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq"] Nov 01 06:27:20.579622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.579585 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9"] Nov 01 06:27:20.580155 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580137 4914 cpu_manager.go:414] "containerMap: removing container" podUID="d95db6e8-52b7-485f-969f-42cc74d6034f" containerName="payload-processing" Nov 01 06:27:20.580155 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580152 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95db6e8-52b7-485f-969f-42cc74d6034f" containerName="payload-processing" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580175 4914 cpu_manager.go:414] "containerMap: removing container" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580183 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580211 4914 cpu_manager.go:414] "containerMap: removing container" podUID="06843d49-e37d-435f-9c33-1d3717789f51" containerName="payload-processing" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580219 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06843d49-e37d-435f-9c33-1d3717789f51" containerName="payload-processing" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580358 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="06843d49-e37d-435f-9c33-1d3717789f51" containerName="payload-processing" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580377 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95db6e8-52b7-485f-969f-42cc74d6034f" containerName="payload-processing" Nov 01 06:27:20.580404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.580385 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fa915f-666d-45ac-9b5f-839e54ac78eb" containerName="istio-proxy" Nov 01 06:27:20.585225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.585210 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.652302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.641528 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b354e3-openshift-default-dockercfg-89zv4\"" Nov 01 06:27:20.711122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711085 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711147 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711204 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711227 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711255 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711309 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711352 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711507 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.711672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.711568 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqcp\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.736954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.734746 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9"] Nov 01 06:27:20.812441 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812408 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812441 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812447 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812638 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812682 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqcp\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812772 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812755 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812790 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812843 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812891 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.812961 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812950 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.813113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.812852 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.813113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.813062 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.813364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.813341 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.813442 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.813420 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.813512 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.813477 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.815353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.815334 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.815567 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.815551 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.869954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.869931 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.870393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.870377 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqcp\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp\") pod \"e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:20.898404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:20.898386 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:21.363369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.363338 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9"] Nov 01 06:27:21.364513 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:27:21.364481 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0265ad_9751_4043_915a_7ccfe6cec445.slice/crio-98dcc5430d6ea08485fb81dd15d1ec1325e18b86e1a840ed37d3e3feaa57a7b3 WatchSource:0}: Error finding container 98dcc5430d6ea08485fb81dd15d1ec1325e18b86e1a840ed37d3e3feaa57a7b3: Status 404 returned error can't find the container with id 98dcc5430d6ea08485fb81dd15d1ec1325e18b86e1a840ed37d3e3feaa57a7b3 Nov 01 06:27:21.366917 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.366884 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:21.367003 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.366969 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:21.367041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.367004 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:21.739348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.738897 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" event={"ID":"bd0265ad-9751-4043-915a-7ccfe6cec445","Type":"ContainerStarted","Data":"32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b"} Nov 01 06:27:21.739348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.739051 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" event={"ID":"bd0265ad-9751-4043-915a-7ccfe6cec445","Type":"ContainerStarted","Data":"98dcc5430d6ea08485fb81dd15d1ec1325e18b86e1a840ed37d3e3feaa57a7b3"} Nov 01 06:27:21.858204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.858146 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podStartSLOduration=1.858126701 podStartE2EDuration="1.858126701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:21.855688013 +0000 UTC m=+1057.408477196" watchObservedRunningTime="2026-11-01 06:27:21.858126701 +0000 UTC m=+1057.410915896" Nov 01 06:27:21.898517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:21.898487 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:22.904531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:22.904503 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:23.803334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:23.800473 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:23.818374 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:23.818350 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:27:24.948022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:24.947988 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" podUID="a91c8651-5499-45dd-af96-f04855dafb5a" containerName="payload-processing" containerID="cri-o://0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229" gracePeriod=25 Nov 01 06:27:25.063133 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.063103 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" podUID="20434eee-bec4-4907-a08b-0733d04b63d6" containerName="payload-pre-processing" containerID="cri-o://5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd" gracePeriod=25 Nov 01 06:27:25.216189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.216136 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:27:25.318262 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.318230 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:27:25.320232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.320214 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume\") pod \"a91c8651-5499-45dd-af96-f04855dafb5a\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " Nov 01 06:27:25.320351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.320327 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g\" (UniqueName: \"kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g\") pod \"a91c8651-5499-45dd-af96-f04855dafb5a\" (UID: \"a91c8651-5499-45dd-af96-f04855dafb5a\") " Nov 01 06:27:25.320524 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.320506 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume" pod "a91c8651-5499-45dd-af96-f04855dafb5a" (UID: "a91c8651-5499-45dd-af96-f04855dafb5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:25.320711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.320698 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a91c8651-5499-45dd-af96-f04855dafb5a-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:25.322407 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.322384 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g" pod "a91c8651-5499-45dd-af96-f04855dafb5a" (UID: "a91c8651-5499-45dd-af96-f04855dafb5a"). InnerVolumeSpecName "kube-api-access-5557g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:25.421956 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.421928 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts\" (UniqueName: \"kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts\") pod \"20434eee-bec4-4907-a08b-0733d04b63d6\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " Nov 01 06:27:25.422105 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.422023 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume\") pod \"20434eee-bec4-4907-a08b-0733d04b63d6\" (UID: \"20434eee-bec4-4907-a08b-0733d04b63d6\") " Nov 01 06:27:25.422294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.422259 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5557g\" (UniqueName: \"kubernetes.io/projected/a91c8651-5499-45dd-af96-f04855dafb5a-kube-api-access-5557g\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:25.422410 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.422391 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume" pod "20434eee-bec4-4907-a08b-0733d04b63d6" (UID: "20434eee-bec4-4907-a08b-0733d04b63d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:25.423963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.423944 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts" pod "20434eee-bec4-4907-a08b-0733d04b63d6" (UID: "20434eee-bec4-4907-a08b-0733d04b63d6"). InnerVolumeSpecName "kube-api-access-lp8ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:25.523507 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.523448 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8ts\" (UniqueName: \"kubernetes.io/projected/20434eee-bec4-4907-a08b-0733d04b63d6-kube-api-access-lp8ts\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:25.523507 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.523470 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20434eee-bec4-4907-a08b-0733d04b63d6-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:25.811173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.811105 4914 generic.go:356] "Generic (PLEG): container finished" podID="a91c8651-5499-45dd-af96-f04855dafb5a" containerID="0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229" exitCode=0 Nov 01 06:27:25.811173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.811143 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" Nov 01 06:27:25.811410 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.811196 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" event={"ID":"a91c8651-5499-45dd-af96-f04855dafb5a","Type":"ContainerDied","Data":"0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229"} Nov 01 06:27:25.811410 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.811249 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw" event={"ID":"a91c8651-5499-45dd-af96-f04855dafb5a","Type":"ContainerDied","Data":"6c784cf2c2d7b07859ccffe0f9f411b41339cc4f6a9901e6aa3f8aaec2e4a350"} Nov 01 06:27:25.811410 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.811304 4914 scope.go:122] "RemoveContainer" containerID="0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229" Nov 01 06:27:25.813082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.813058 4914 generic.go:356] "Generic (PLEG): container finished" podID="20434eee-bec4-4907-a08b-0733d04b63d6" containerID="5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd" exitCode=0 Nov 01 06:27:25.813202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.813091 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" Nov 01 06:27:25.813202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.813085 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" event={"ID":"20434eee-bec4-4907-a08b-0733d04b63d6","Type":"ContainerDied","Data":"5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd"} Nov 01 06:27:25.813341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.813227 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq" event={"ID":"20434eee-bec4-4907-a08b-0733d04b63d6","Type":"ContainerDied","Data":"0d2ffac248ef6375a52a6ab63c39f7c305bf5bc3a09002ff3ecbf9c14d6f213b"} Nov 01 06:27:25.821120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.821101 4914 scope.go:122] "RemoveContainer" containerID="0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229" Nov 01 06:27:25.821454 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:27:25.821429 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229\": container with ID starting with 0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229 not found: ID does not exist" containerID="0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229" Nov 01 06:27:25.821543 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.821465 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229"} err="failed to get container status \"0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229\": rpc error: code = NotFound desc = could not find container \"0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229\": container with ID starting with 0a385d2860c494d6de5f26a093cb6914c412fe0d9ef36548a3b22de1ec414229 not found: ID does not exist" Nov 01 06:27:25.821543 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.821487 4914 scope.go:122] "RemoveContainer" containerID="5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd" Nov 01 06:27:25.829680 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.829665 4914 scope.go:122] "RemoveContainer" containerID="5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd" Nov 01 06:27:25.829883 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:27:25.829863 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd\": container with ID starting with 5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd not found: ID does not exist" containerID="5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd" Nov 01 06:27:25.829942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.829892 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd"} err="failed to get container status \"5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd\": rpc error: code = NotFound desc = could not find container \"5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd\": container with ID starting with 5260c70110a537afc7ebbcd7d18e53ecb59f1e3cf2b8d3a84fe05136f356bedd not found: ID does not exist" Nov 01 06:27:25.991129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.991096 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq"] Nov 01 06:27:25.993473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:25.993448 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd9d24-85ff89f5-l5ddq"] Nov 01 06:27:26.098851 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:26.098826 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw"] Nov 01 06:27:26.105148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:26.103524 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd9d24-56cd9d9699-vp9lw"] Nov 01 06:27:26.616518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:26.616492 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20434eee-bec4-4907-a08b-0733d04b63d6" path="/var/lib/kubelet/pods/20434eee-bec4-4907-a08b-0733d04b63d6/volumes" Nov 01 06:27:26.616841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:26.616830 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91c8651-5499-45dd-af96-f04855dafb5a" path="/var/lib/kubelet/pods/a91c8651-5499-45dd-af96-f04855dafb5a/volumes" Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.158937 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5"] Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.159780 4914 cpu_manager.go:414] "containerMap: removing container" podUID="a91c8651-5499-45dd-af96-f04855dafb5a" containerName="payload-processing" Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.159798 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91c8651-5499-45dd-af96-f04855dafb5a" containerName="payload-processing" Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.159815 4914 cpu_manager.go:414] "containerMap: removing container" podUID="20434eee-bec4-4907-a08b-0733d04b63d6" containerName="payload-pre-processing" Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.159823 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20434eee-bec4-4907-a08b-0733d04b63d6" containerName="payload-pre-processing" Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.159974 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91c8651-5499-45dd-af96-f04855dafb5a" containerName="payload-processing" Nov 01 06:27:29.169300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.159986 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="20434eee-bec4-4907-a08b-0733d04b63d6" containerName="payload-pre-processing" Nov 01 06:27:29.211618 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.209886 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.228381 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.228339 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5"] Nov 01 06:27:29.268232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.268201 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b354e3\"" Nov 01 06:27:29.268232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.268208 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b354e3-dockercfg-966r5\"" Nov 01 06:27:29.272824 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.272785 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgvn\" (UniqueName: \"kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn\") pod \"payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.272965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.272924 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.320411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.320385 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n"] Nov 01 06:27:29.324534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.324515 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.373439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.373416 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.373526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.373447 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mhs\" (UniqueName: \"kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.373526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.373516 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgvn\" (UniqueName: \"kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn\") pod \"payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.373609 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.373565 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.374199 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.374181 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.378655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.378634 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n"] Nov 01 06:27:29.429651 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.429596 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgvn\" (UniqueName: \"kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn\") pod \"payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.474347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.474325 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.474430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.474387 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mhs\" (UniqueName: \"kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.475008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.474988 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.530541 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.530515 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mhs\" (UniqueName: \"kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs\") pod \"payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.555627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.555601 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:29.638308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.637532 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:29.927246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:29.927219 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5"] Nov 01 06:27:29.928829 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:27:29.928805 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3d3a0e_b97e_4cb4_bc04_bc5550c79182.slice/crio-f9bd932819ba695260a64fa81633428acf8bac7a4701a22c75fd68200ed9d231 WatchSource:0}: Error finding container f9bd932819ba695260a64fa81633428acf8bac7a4701a22c75fd68200ed9d231: Status 404 returned error can't find the container with id f9bd932819ba695260a64fa81633428acf8bac7a4701a22c75fd68200ed9d231 Nov 01 06:27:30.028642 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.028620 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n"] Nov 01 06:27:30.029197 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:27:30.029177 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24379c49_137f_43a6_a40f_d53c32d1ab8d.slice/crio-1c7396d67500f0630bb15200763f1dc2381556067718d797d90deff6a553ce4b WatchSource:0}: Error finding container 1c7396d67500f0630bb15200763f1dc2381556067718d797d90deff6a553ce4b: Status 404 returned error can't find the container with id 1c7396d67500f0630bb15200763f1dc2381556067718d797d90deff6a553ce4b Nov 01 06:27:30.843911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.843878 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" event={"ID":"24379c49-137f-43a6-a40f-d53c32d1ab8d","Type":"ContainerStarted","Data":"57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4"} Nov 01 06:27:30.844322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.843940 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:30.844322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.843963 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" event={"ID":"24379c49-137f-43a6-a40f-d53c32d1ab8d","Type":"ContainerStarted","Data":"1c7396d67500f0630bb15200763f1dc2381556067718d797d90deff6a553ce4b"} Nov 01 06:27:30.845490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.845463 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" event={"ID":"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182","Type":"ContainerStarted","Data":"4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a"} Nov 01 06:27:30.845597 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.845505 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" event={"ID":"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182","Type":"ContainerStarted","Data":"f9bd932819ba695260a64fa81633428acf8bac7a4701a22c75fd68200ed9d231"} Nov 01 06:27:30.845597 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.845531 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:30.967419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:30.961800 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" podStartSLOduration=1.9617692949999999 podStartE2EDuration="1.961769295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:30.958980824 +0000 UTC m=+1066.511770063" watchObservedRunningTime="2026-11-01 06:27:30.961769295 +0000 UTC m=+1066.514558490" Nov 01 06:27:31.091433 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:31.091367 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" podStartSLOduration=2.091344105 podStartE2EDuration="2.091344105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:31.078722605 +0000 UTC m=+1066.631511798" watchObservedRunningTime="2026-11-01 06:27:31.091344105 +0000 UTC m=+1066.644133298" Nov 01 06:27:37.592101 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:27:37.592065 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:27:41.854173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:41.854143 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:27:41.854708 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:41.854227 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:27:49.564851 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.564820 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m"] Nov 01 06:27:49.565234 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.565029 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" podUID="adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" containerName="maas-api" containerID="cri-o://87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf" gracePeriod=30 Nov 01 06:27:49.878605 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.878580 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:27:49.969190 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.969165 4914 generic.go:356] "Generic (PLEG): container finished" podID="adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" containerID="87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf" exitCode=0 Nov 01 06:27:49.969361 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.969206 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" Nov 01 06:27:49.969361 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.969259 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" event={"ID":"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92","Type":"ContainerDied","Data":"87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf"} Nov 01 06:27:49.969361 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.969346 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m" event={"ID":"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92","Type":"ContainerDied","Data":"6af0f845a632e24954448dd267092b8ee24e886a4341476cbba2e6a6b1248c2e"} Nov 01 06:27:49.969493 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.969377 4914 scope.go:122] "RemoveContainer" containerID="87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf" Nov 01 06:27:49.979142 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.979126 4914 scope.go:122] "RemoveContainer" containerID="87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf" Nov 01 06:27:49.979420 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:27:49.979401 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf\": container with ID starting with 87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf not found: ID does not exist" containerID="87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf" Nov 01 06:27:49.979490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.979426 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf"} err="failed to get container status \"87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf\": rpc error: code = NotFound desc = could not find container \"87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf\": container with ID starting with 87e76f5557f86a972d13c75af84fe509cbbe366ae96653445936ee0bc3ac42cf not found: ID does not exist" Nov 01 06:27:49.993224 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.993204 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs\") pod \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " Nov 01 06:27:49.993371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.993250 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87\" (UniqueName: \"kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87\") pod \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " Nov 01 06:27:49.993436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.993369 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca\") pod \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " Nov 01 06:27:49.993436 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.993409 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\") pod \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\" (UID: \"adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92\") " Nov 01 06:27:49.993990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.993944 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca" pod "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" (UID: "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:27:49.996545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.996516 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87" pod "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" (UID: "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92"). InnerVolumeSpecName "kube-api-access-5mv87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:27:49.996660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.996547 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls" pod "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" (UID: "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:27:49.998656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:49.998610 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs" pod "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" (UID: "adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:27:50.094568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.094516 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mv87\" (UniqueName: \"kubernetes.io/projected/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-kube-api-access-5mv87\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:50.094568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.094540 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:50.094568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.094550 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:50.094568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.094559 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:27:50.394413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.394353 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m"] Nov 01 06:27:50.395915 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.395889 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-85584b-5c79c49db9-bnn4m"] Nov 01 06:27:50.616925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:50.616860 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" path="/var/lib/kubelet/pods/adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92/volumes" Nov 01 06:27:57.626445 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.626409 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2"] Nov 01 06:27:57.629028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.626941 4914 cpu_manager.go:414] "containerMap: removing container" podUID="adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" containerName="maas-api" Nov 01 06:27:57.629028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.626954 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" containerName="maas-api" Nov 01 06:27:57.629028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.627065 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfc3b19-8c3a-4a9f-9b5a-953c7b3a4a92" containerName="maas-api" Nov 01 06:27:57.630187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.630173 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.679297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.678575 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49"] Nov 01 06:27:57.692296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.683318 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.703910 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.703964 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704000 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704027 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704055 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6tq\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704081 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704116 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704145 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704208 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704231 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704298 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704328 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704374 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704424 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704470 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704529 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704557 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.705294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.704579 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.731399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.731375 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2"] Nov 01 06:27:57.825310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825250 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825521 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825560 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6tq\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825588 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825628 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825659 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825724 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825748 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825792 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825820 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825860 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825905 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.825950 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.826005 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.826032 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.826055 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.826123 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.826309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.826161 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.827414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.827127 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.827501 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.827430 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.828423 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.827838 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.828423 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.827936 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.828423 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.828062 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.828423 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.828201 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.828860 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.828833 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.830751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.829948 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.832950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.832919 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.834126 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.834101 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.836009 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.835916 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.838165 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.838030 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.838773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.838721 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.839183 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.839164 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.841077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.841058 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49"] Nov 01 06:27:57.920372 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.914122 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.926326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.925717 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.937327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.929258 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:57.937327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.929263 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6tq\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:57.944238 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:57.944055 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:58.030016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.029982 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:27:58.488627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.488597 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2"] Nov 01 06:27:58.490020 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:27:58.489996 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932b4994_7d4c_4b6a_9832_c6a179828959.slice/crio-ce4663bb997db99b3c878bb55d7211b21ce432ab605a625ddf2b3269072e8a54 WatchSource:0}: Error finding container ce4663bb997db99b3c878bb55d7211b21ce432ab605a625ddf2b3269072e8a54: Status 404 returned error can't find the container with id ce4663bb997db99b3c878bb55d7211b21ce432ab605a625ddf2b3269072e8a54 Nov 01 06:27:58.492230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.492197 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:58.492330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.492263 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:58.492330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.492323 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:58.540140 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.540115 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49"] Nov 01 06:27:58.541438 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:27:58.541411 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f842fa_4ca0_4804_af3e_c84ba34bf6aa.slice/crio-d4ddff270a87d30d607ca41b69baa0a4d484b69088f31358cd5859c784955766 WatchSource:0}: Error finding container d4ddff270a87d30d607ca41b69baa0a4d484b69088f31358cd5859c784955766: Status 404 returned error can't find the container with id d4ddff270a87d30d607ca41b69baa0a4d484b69088f31358cd5859c784955766 Nov 01 06:27:58.543739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.543710 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:58.543838 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.543768 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:58.543838 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:58.543797 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:27:59.140306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.140241 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" event={"ID":"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa","Type":"ContainerStarted","Data":"55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c"} Nov 01 06:27:59.140735 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.140327 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" event={"ID":"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa","Type":"ContainerStarted","Data":"d4ddff270a87d30d607ca41b69baa0a4d484b69088f31358cd5859c784955766"} Nov 01 06:27:59.143791 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.143755 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" event={"ID":"932b4994-7d4c-4b6a-9832-c6a179828959","Type":"ContainerStarted","Data":"15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037"} Nov 01 06:27:59.143942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.143814 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" event={"ID":"932b4994-7d4c-4b6a-9832-c6a179828959","Type":"ContainerStarted","Data":"ce4663bb997db99b3c878bb55d7211b21ce432ab605a625ddf2b3269072e8a54"} Nov 01 06:27:59.259233 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.259181 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podStartSLOduration=2.259165318 podStartE2EDuration="2.259165318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:59.255464293 +0000 UTC m=+1094.808253479" watchObservedRunningTime="2026-11-01 06:27:59.259165318 +0000 UTC m=+1094.811954511" Nov 01 06:27:59.375000 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.374954 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podStartSLOduration=2.374940507 podStartE2EDuration="2.374940507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:27:59.369257751 +0000 UTC m=+1094.922046943" watchObservedRunningTime="2026-11-01 06:27:59.374940507 +0000 UTC m=+1094.927729698" Nov 01 06:27:59.944828 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.944664 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:27:59.983308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.963604 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:27:59.983308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:27:59.963670 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 06:28:00.030940 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:00.030911 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:28:00.032109 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:00.032088 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:28:00.032179 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:00.032141 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:28:00.945149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:00.945109 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:28:00.945696 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:00.945165 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 06:28:01.030376 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:01.030343 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:28:01.030535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:01.030392 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:28:01.945670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:01.945622 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:28:01.946248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:01.946221 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 06:28:02.037317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:02.034653 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:28:02.037317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:02.034714 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:28:02.255354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:02.245732 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h"] Nov 01 06:28:02.944399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:02.944365 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:28:02.944612 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:02.944438 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 06:28:03.030961 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:03.030924 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:28:03.031348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:03.030981 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:28:03.955299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:03.947481 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:28:03.955299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:03.947534 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 06:28:04.053300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:04.043266 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:28:04.053300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:04.043355 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:28:04.945022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:04.944985 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 01 06:28:04.945189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:04.945054 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 01 06:28:05.030787 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:05.030748 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 01 06:28:05.030948 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:05.030822 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 01 06:28:06.024245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:06.024200 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:28:06.024763 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:06.024653 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:28:06.026210 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:06.026184 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:28:06.387621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:06.387581 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:28:06.388932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:06.388904 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:28:06.390092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:06.390072 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:28:07.246159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:07.246126 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerName="payload-pre-processing" containerID="cri-o://7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514" gracePeriod=25 Nov 01 06:28:07.591960 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:07.591884 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:28:07.843723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:07.843641 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:28:09.203691 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.203659 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr"] Nov 01 06:28:09.207719 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.207704 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.244271 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.244247 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.244397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.244310 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.244460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.244440 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q78z\" (UniqueName: \"kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.244506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.244484 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.261253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.261225 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b354e3-serving-cert\"" Nov 01 06:28:09.314806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.314786 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr"] Nov 01 06:28:09.345787 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.345759 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.345925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.345814 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.345925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.345881 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q78z\" (UniqueName: \"kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.346050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.345929 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.346050 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:09.345995 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-serving-cert: secret "maas-api-e2e-ipp-b-b354e3-serving-cert" not found Nov 01 06:28:09.346142 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:09.346075 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls podName:c50a8686-d2a1-4302-bdac-61715a6e77d8 nodeName:}" failed. No retries permitted until 2026-11-01 06:28:09.846054075 +0000 UTC m=+1105.398843269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls") pod "maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" (UID: "c50a8686-d2a1-4302-bdac-61715a6e77d8") : secret "maas-api-e2e-ipp-b-b354e3-serving-cert" not found Nov 01 06:28:09.346734 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.346712 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.348817 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.348795 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.402579 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.402546 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q78z\" (UniqueName: \"kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.852405 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.852371 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:09.854930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:09.854904 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:10.121319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:10.121254 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:10.490363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:10.490338 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr"] Nov 01 06:28:10.491241 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:28:10.491215 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50a8686_d2a1_4302_bdac_61715a6e77d8.slice/crio-cff418b3967510da93ed8eb70a5d0034f4397a4a4252140eb8688291f804fa12 WatchSource:0}: Error finding container cff418b3967510da93ed8eb70a5d0034f4397a4a4252140eb8688291f804fa12: Status 404 returned error can't find the container with id cff418b3967510da93ed8eb70a5d0034f4397a4a4252140eb8688291f804fa12 Nov 01 06:28:11.277439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:11.277401 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" event={"ID":"c50a8686-d2a1-4302-bdac-61715a6e77d8","Type":"ContainerStarted","Data":"cff418b3967510da93ed8eb70a5d0034f4397a4a4252140eb8688291f804fa12"} Nov 01 06:28:12.285112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:12.285068 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" event={"ID":"c50a8686-d2a1-4302-bdac-61715a6e77d8","Type":"ContainerStarted","Data":"b8fa2a0a0aa172e4b5637be28ba7fc196f11bd4a8109d94f8097e493358169cc"} Nov 01 06:28:12.285554 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:12.285258 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:12.400407 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:12.400363 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" podStartSLOduration=2.4992487199999998 podStartE2EDuration="3.400350534s" totalImagesPullingTime="901.101814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:12.398249978 +0000 UTC m=+1107.951039169" watchObservedRunningTime="2026-11-01 06:28:12.400350534 +0000 UTC m=+1107.953139725" Nov 01 06:28:13.849312 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:13.811212 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx"] Nov 01 06:28:13.849312 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:13.811535 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" podUID="caf97c07-4c8d-49c2-9d6c-45d6306d9d40" containerName="maas-api" containerID="cri-o://54c87f43bc73c7f3723aa9ee0b7f83892bf62add2f5491848deb2ef37ccb87e4" gracePeriod=30 Nov 01 06:28:15.458533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.458227 4914 generic.go:356] "Generic (PLEG): container finished" podID="caf97c07-4c8d-49c2-9d6c-45d6306d9d40" containerID="54c87f43bc73c7f3723aa9ee0b7f83892bf62add2f5491848deb2ef37ccb87e4" exitCode=0 Nov 01 06:28:15.458533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.458428 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" event={"ID":"caf97c07-4c8d-49c2-9d6c-45d6306d9d40","Type":"ContainerDied","Data":"54c87f43bc73c7f3723aa9ee0b7f83892bf62add2f5491848deb2ef37ccb87e4"} Nov 01 06:28:15.458533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.458483 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" event={"ID":"caf97c07-4c8d-49c2-9d6c-45d6306d9d40","Type":"ContainerDied","Data":"ba11405630661957daaf768d23105f731a8cab861f8c41261c7840177630f775"} Nov 01 06:28:15.458533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.458507 4914 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba11405630661957daaf768d23105f731a8cab861f8c41261c7840177630f775" Nov 01 06:28:15.487589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.487210 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:28:15.549691 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.549661 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs\") pod \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " Nov 01 06:28:15.549879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.549707 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\") pod \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " Nov 01 06:28:15.549879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.549738 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll\" (UniqueName: \"kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll\") pod \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " Nov 01 06:28:15.549879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.549820 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca\") pod \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\" (UID: \"caf97c07-4c8d-49c2-9d6c-45d6306d9d40\") " Nov 01 06:28:15.550268 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.550242 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca" pod "caf97c07-4c8d-49c2-9d6c-45d6306d9d40" (UID: "caf97c07-4c8d-49c2-9d6c-45d6306d9d40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:15.553390 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.553369 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls" pod "caf97c07-4c8d-49c2-9d6c-45d6306d9d40" (UID: "caf97c07-4c8d-49c2-9d6c-45d6306d9d40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:15.553508 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.553388 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs" pod "caf97c07-4c8d-49c2-9d6c-45d6306d9d40" (UID: "caf97c07-4c8d-49c2-9d6c-45d6306d9d40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:28:15.553508 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.553421 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll" pod "caf97c07-4c8d-49c2-9d6c-45d6306d9d40" (UID: "caf97c07-4c8d-49c2-9d6c-45d6306d9d40"). InnerVolumeSpecName "kube-api-access-crjll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:15.650871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.650846 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:15.650871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.650871 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:15.651039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.650882 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjll\" (UniqueName: \"kubernetes.io/projected/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-kube-api-access-crjll\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:15.651039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:15.650892 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf97c07-4c8d-49c2-9d6c-45d6306d9d40-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:16.463590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:16.463561 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx" Nov 01 06:28:16.625746 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:16.625702 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx"] Nov 01 06:28:16.675382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:16.675352 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef9b55-867d886c56-dbzvx"] Nov 01 06:28:17.844699 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:17.844668 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:28:18.305945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:18.305916 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:28:18.617821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:18.617790 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf97c07-4c8d-49c2-9d6c-45d6306d9d40" path="/var/lib/kubelet/pods/caf97c07-4c8d-49c2-9d6c-45d6306d9d40/volumes" Nov 01 06:28:23.782080 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:23.782045 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw"] Nov 01 06:28:23.782599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:23.782315 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" containerID="cri-o://dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673" gracePeriod=30 Nov 01 06:28:27.507049 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.506988 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:28:27.541263 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.541221 4914 generic.go:356] "Generic (PLEG): container finished" podID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerID="7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514" exitCode=0 Nov 01 06:28:27.541417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.541269 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" event={"ID":"3583442f-cf21-4bd8-884d-ec9ef8fff3c2","Type":"ContainerDied","Data":"7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514"} Nov 01 06:28:27.541417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.541319 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" Nov 01 06:28:27.541535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.541461 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h" event={"ID":"3583442f-cf21-4bd8-884d-ec9ef8fff3c2","Type":"ContainerDied","Data":"e7ad85795cf24854402c71bab5432e15a70e79888c05dd00c85c5d702de62baa"} Nov 01 06:28:27.541535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.541491 4914 scope.go:122] "RemoveContainer" containerID="7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514" Nov 01 06:28:27.550666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.550645 4914 scope.go:122] "RemoveContainer" containerID="7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514" Nov 01 06:28:27.550947 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:27.550918 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514\": container with ID starting with 7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514 not found: ID does not exist" containerID="7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514" Nov 01 06:28:27.551030 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.550954 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514"} err="failed to get container status \"7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514\": rpc error: code = NotFound desc = could not find container \"7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514\": container with ID starting with 7344efb7b7f16acd4a295b4fd54dc4e139d49206f43245a79521d50dccc81514 not found: ID does not exist" Nov 01 06:28:27.593727 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.593709 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume\") pod \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " Nov 01 06:28:27.593831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.593750 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss\" (UniqueName: \"kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss\") pod \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\" (UID: \"3583442f-cf21-4bd8-884d-ec9ef8fff3c2\") " Nov 01 06:28:27.594119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.594096 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume" pod "3583442f-cf21-4bd8-884d-ec9ef8fff3c2" (UID: "3583442f-cf21-4bd8-884d-ec9ef8fff3c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:27.594235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.594221 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:27.595764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.595743 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss" pod "3583442f-cf21-4bd8-884d-ec9ef8fff3c2" (UID: "3583442f-cf21-4bd8-884d-ec9ef8fff3c2"). InnerVolumeSpecName "kube-api-access-rrwss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:27.695897 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.695817 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwss\" (UniqueName: \"kubernetes.io/projected/3583442f-cf21-4bd8-884d-ec9ef8fff3c2-kube-api-access-rrwss\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:27.963830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.963803 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h"] Nov 01 06:28:27.967056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:27.967031 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ef9b55-7bdbc7ff79-p5k8h"] Nov 01 06:28:28.617391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:28.617358 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" path="/var/lib/kubelet/pods/3583442f-cf21-4bd8-884d-ec9ef8fff3c2/volumes" Nov 01 06:28:29.049031 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.049010 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:28:29.109297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109255 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109340 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109385 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109421 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109502 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109471 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109543 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109526 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:28:29.109604 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109590 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109629 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109714 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109686 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109771 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109730 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs\") pod \"53cd4519-8810-4b9a-bc09-08214020d1f3\" (UID: \"53cd4519-8810-4b9a-bc09-08214020d1f3\") " Nov 01 06:28:29.109886 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.109829 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:28:29.110040 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.110020 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:28:29.110134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.110116 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.110193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.110092 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:28:29.110193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.110165 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.110193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.110183 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.110715 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.110690 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:28:29.111855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.111829 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:28:29.111942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.111875 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:29.112096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.112077 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "kube-api-access-5b2p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:28:29.112254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.112239 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy" pod "53cd4519-8810-4b9a-bc09-08214020d1f3" (UID: "53cd4519-8810-4b9a-bc09-08214020d1f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:28:29.210952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.210904 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53cd4519-8810-4b9a-bc09-08214020d1f3-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.210952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.210926 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.210952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.210934 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.210952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.210942 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.210952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.210951 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.211144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.210959 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b2p5\" (UniqueName: \"kubernetes.io/projected/53cd4519-8810-4b9a-bc09-08214020d1f3-kube-api-access-5b2p5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:28:29.555326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.555223 4914 generic.go:356] "Generic (PLEG): container finished" podID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerID="dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673" exitCode=0 Nov 01 06:28:29.555326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.555307 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" Nov 01 06:28:29.555488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.555319 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" event={"ID":"53cd4519-8810-4b9a-bc09-08214020d1f3","Type":"ContainerDied","Data":"dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673"} Nov 01 06:28:29.555488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.555374 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw" event={"ID":"53cd4519-8810-4b9a-bc09-08214020d1f3","Type":"ContainerDied","Data":"ee40c0fd12b254eaab8f83ddbd959035d51cadcf80a79c49464072f5b843b6fb"} Nov 01 06:28:29.555488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.555395 4914 scope.go:122] "RemoveContainer" containerID="dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673" Nov 01 06:28:29.566083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.566069 4914 scope.go:122] "RemoveContainer" containerID="da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603" Nov 01 06:28:29.574348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.574325 4914 scope.go:122] "RemoveContainer" containerID="dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673" Nov 01 06:28:29.574599 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:29.574578 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673\": container with ID starting with dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673 not found: ID does not exist" containerID="dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673" Nov 01 06:28:29.574638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.574607 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673"} err="failed to get container status \"dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673\": rpc error: code = NotFound desc = could not find container \"dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673\": container with ID starting with dfb0461a92e385192014df3e2efbbdeb918cc25db2739ded96da9bedc977e673 not found: ID does not exist" Nov 01 06:28:29.574638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.574623 4914 scope.go:122] "RemoveContainer" containerID="da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603" Nov 01 06:28:29.574867 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:29.574852 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603\": container with ID starting with da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603 not found: ID does not exist" containerID="da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603" Nov 01 06:28:29.574911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.574873 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603"} err="failed to get container status \"da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603\": rpc error: code = NotFound desc = could not find container \"da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603\": container with ID starting with da59b6aa0d99420d73aa729afcfc4a4ab482d22f4611f2848e73ba4767dcb603 not found: ID does not exist" Nov 01 06:28:29.717652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.717629 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw"] Nov 01 06:28:29.768256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:29.768238 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef9b55-openshift-default-6ccbb7664f-h67kw"] Nov 01 06:28:30.621703 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:30.621671 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" path="/var/lib/kubelet/pods/53cd4519-8810-4b9a-bc09-08214020d1f3/volumes" Nov 01 06:28:33.629250 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.629215 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk"] Nov 01 06:28:33.630115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.630090 4914 cpu_manager.go:414] "containerMap: removing container" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" Nov 01 06:28:33.630228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.630117 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" Nov 01 06:28:33.630228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.630141 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerName="payload-pre-processing" Nov 01 06:28:33.630228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.630148 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerName="payload-pre-processing" Nov 01 06:28:33.630228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.630196 4914 cpu_manager.go:414] "containerMap: removing container" podUID="caf97c07-4c8d-49c2-9d6c-45d6306d9d40" containerName="maas-api" Nov 01 06:28:33.630228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.630204 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf97c07-4c8d-49c2-9d6c-45d6306d9d40" containerName="maas-api" Nov 01 06:28:33.633400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.632593 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" Nov 01 06:28:33.633400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.632622 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3583442f-cf21-4bd8-884d-ec9ef8fff3c2" containerName="payload-pre-processing" Nov 01 06:28:33.633400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.632636 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" Nov 01 06:28:33.633400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.632654 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf97c07-4c8d-49c2-9d6c-45d6306d9d40" containerName="maas-api" Nov 01 06:28:33.668036 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.653517 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk"] Nov 01 06:28:33.668036 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.654385 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.703734 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.703704 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h44\" (UniqueName: \"kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44\") pod \"payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.703913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.703890 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.770640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.770587 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e7b2a3-dockercfg-wgw28\"" Nov 01 06:28:33.771264 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.771243 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e7b2a3\"" Nov 01 06:28:33.805643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.805621 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h44\" (UniqueName: \"kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44\") pod \"payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.805752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.805687 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.806371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.806354 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.872591 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.872562 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h44\" (UniqueName: \"kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44\") pod \"payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:33.989202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:33.989133 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:34.375031 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:34.374992 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk"] Nov 01 06:28:34.375183 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:28:34.375161 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747bf3f0_a5f9_4f6c_bf68_9e55849714a9.slice/crio-66196f80533d2f504ce20b739d8a7a0140a60538a3ceded415a46a4fcbeb230d WatchSource:0}: Error finding container 66196f80533d2f504ce20b739d8a7a0140a60538a3ceded415a46a4fcbeb230d: Status 404 returned error can't find the container with id 66196f80533d2f504ce20b739d8a7a0140a60538a3ceded415a46a4fcbeb230d Nov 01 06:28:34.661353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:34.655157 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" event={"ID":"747bf3f0-a5f9-4f6c-bf68-9e55849714a9","Type":"ContainerStarted","Data":"fdf926a005a1faf40aaca9e206b2bb2fbf03d5e354150f7f32dd5dc93bb40624"} Nov 01 06:28:34.661353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:34.655216 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" event={"ID":"747bf3f0-a5f9-4f6c-bf68-9e55849714a9","Type":"ContainerStarted","Data":"66196f80533d2f504ce20b739d8a7a0140a60538a3ceded415a46a4fcbeb230d"} Nov 01 06:28:34.661353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:34.655923 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:34.775099 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:34.775039 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" podStartSLOduration=1.775020795 podStartE2EDuration="1.775020795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:34.771089347 +0000 UTC m=+1130.323878540" watchObservedRunningTime="2026-11-01 06:28:34.775020795 +0000 UTC m=+1130.327809988" Nov 01 06:28:37.591545 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:28:37.591495 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:28:39.840447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:39.840411 4914 patch_prober.go:29] interesting pod/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:28:39.840926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:39.840474 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:28:40.703152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:40.703123 4914 generic.go:356] "Generic (PLEG): container finished" podID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerID="32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b" exitCode=0 Nov 01 06:28:40.703367 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:40.703171 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" event={"ID":"bd0265ad-9751-4043-915a-7ccfe6cec445","Type":"ContainerDied","Data":"32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b"} Nov 01 06:28:40.703484 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:40.703469 4914 scope.go:122] "RemoveContainer" containerID="32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b" Nov 01 06:28:40.704550 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:40.704521 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:28:40.704639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:40.704584 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:28:40.704639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:40.704612 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:28:41.711330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:41.711263 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" event={"ID":"bd0265ad-9751-4043-915a-7ccfe6cec445","Type":"ContainerStarted","Data":"366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1"} Nov 01 06:28:41.899339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:41.899304 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:28:41.899530 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:41.899422 4914 patch_prober.go:29] interesting pod/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 06:28:41.899530 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:41.899475 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 06:28:42.898871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:42.898840 4914 patch_prober.go:29] interesting pod/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 06:28:42.899257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:42.898895 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 06:28:43.019370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.019337 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr"] Nov 01 06:28:43.019961 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.019938 4914 cpu_manager.go:414] "containerMap: removing container" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" Nov 01 06:28:43.019961 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.019958 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cd4519-8810-4b9a-bc09-08214020d1f3" containerName="istio-proxy" Nov 01 06:28:43.024119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.024098 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.125313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.123111 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr"] Nov 01 06:28:43.133982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.133952 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134128 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.133995 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134128 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134101 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134197 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134140 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134240 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134359 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134425 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134474 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134464 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96vc\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.134526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.134494 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.235873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.235800 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.235873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.235845 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96vc\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.235876 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.235969 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.235992 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236036 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236080 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236116 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236194 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236234 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236397 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236495 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236507 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.236817 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.236800 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.238206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.238189 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.238609 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.238593 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.292082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.292052 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.292217 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.292200 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96vc\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.338153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.338132 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:43.786540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.786507 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr"] Nov 01 06:28:43.787831 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:28:43.787810 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e78e54_59de_4cad_ad4e_dda46617a274.slice/crio-7c430a8a811e4b4bf50858d601c95cfbc9c133a5dc0a108550c6a57d3b8dfddf WatchSource:0}: Error finding container 7c430a8a811e4b4bf50858d601c95cfbc9c133a5dc0a108550c6a57d3b8dfddf: Status 404 returned error can't find the container with id 7c430a8a811e4b4bf50858d601c95cfbc9c133a5dc0a108550c6a57d3b8dfddf Nov 01 06:28:43.790364 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.790328 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:28:43.790439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.790397 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:28:43.790479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.790434 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:28:43.898849 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.898819 4914 patch_prober.go:29] interesting pod/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 06:28:43.898974 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:43.898874 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 06:28:44.729832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:44.729794 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" event={"ID":"65e78e54-59de-4cad-ad4e-dda46617a274","Type":"ContainerStarted","Data":"8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59"} Nov 01 06:28:44.730052 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:44.729847 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" event={"ID":"65e78e54-59de-4cad-ad4e-dda46617a274","Type":"ContainerStarted","Data":"7c430a8a811e4b4bf50858d601c95cfbc9c133a5dc0a108550c6a57d3b8dfddf"} Nov 01 06:28:44.818127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:44.818096 4914 scope.go:122] "RemoveContainer" containerID="54c87f43bc73c7f3723aa9ee0b7f83892bf62add2f5491848deb2ef37ccb87e4" Nov 01 06:28:44.845223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:44.845183 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podStartSLOduration=2.8451697879999998 podStartE2EDuration="2.845169788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:28:44.844590725 +0000 UTC m=+1140.397379919" watchObservedRunningTime="2026-11-01 06:28:44.845169788 +0000 UTC m=+1140.397959037" Nov 01 06:28:44.900710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:44.900674 4914 patch_prober.go:29] interesting pod/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 06:28:44.901322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:44.901265 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 06:28:45.339206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:45.339176 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:45.340433 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:45.340416 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:45.340491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:45.340454 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:45.898832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:45.898794 4914 patch_prober.go:29] interesting pod/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 01 06:28:45.899002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:45.898839 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 01 06:28:46.338755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:46.338668 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:46.338755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:46.338732 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:46.683850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:46.683817 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:28:46.902665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:46.902633 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:28:46.902994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:46.902974 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:28:46.903649 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:46.903623 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:28:47.338661 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:47.338623 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:47.338854 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:47.338689 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:48.349294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:48.347689 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:48.349294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:48.347755 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:49.340298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:49.339788 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:49.340298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:49.339869 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:50.346301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:50.339923 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:50.346301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:50.340001 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:51.338652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:51.338620 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 01 06:28:51.338817 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:51.338660 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 01 06:28:52.914822 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:52.914790 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:52.915864 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:52.915844 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:28:52.916495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:28:52.916475 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:29:01.502827 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:01.502795 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5"] Nov 01 06:29:01.584576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:01.584545 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n"] Nov 01 06:29:06.503941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:06.503881 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" podUID="ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" containerName="payload-processing" containerID="cri-o://4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a" gracePeriod=25 Nov 01 06:29:06.584964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:06.584927 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" podUID="24379c49-137f-43a6-a40f-d53c32d1ab8d" containerName="payload-pre-processing" containerID="cri-o://57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4" gracePeriod=25 Nov 01 06:29:06.847499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:06.847477 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:29:06.869865 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:06.869846 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:29:07.021583 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.021553 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume\") pod \"24379c49-137f-43a6-a40f-d53c32d1ab8d\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " Nov 01 06:29:07.021724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.021588 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn\" (UniqueName: \"kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn\") pod \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " Nov 01 06:29:07.021724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.021645 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume\") pod \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\" (UID: \"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182\") " Nov 01 06:29:07.021724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.021699 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs\" (UniqueName: \"kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs\") pod \"24379c49-137f-43a6-a40f-d53c32d1ab8d\" (UID: \"24379c49-137f-43a6-a40f-d53c32d1ab8d\") " Nov 01 06:29:07.021977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.021945 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume" pod "24379c49-137f-43a6-a40f-d53c32d1ab8d" (UID: "24379c49-137f-43a6-a40f-d53c32d1ab8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:29:07.022078 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.022039 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume" pod "ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" (UID: "ae3d3a0e-b97e-4cb4-bc04-bc5550c79182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:29:07.022164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.022152 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24379c49-137f-43a6-a40f-d53c32d1ab8d-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:07.022203 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.022167 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:07.023902 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.023882 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs" pod "24379c49-137f-43a6-a40f-d53c32d1ab8d" (UID: "24379c49-137f-43a6-a40f-d53c32d1ab8d"). InnerVolumeSpecName "kube-api-access-t5mhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:07.023970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.023911 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn" pod "ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" (UID: "ae3d3a0e-b97e-4cb4-bc04-bc5550c79182"). InnerVolumeSpecName "kube-api-access-8hgvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:29:07.051725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.051672 4914 generic.go:356] "Generic (PLEG): container finished" podID="ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" containerID="4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a" exitCode=0 Nov 01 06:29:07.051725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.051713 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" Nov 01 06:29:07.051834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.051753 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" event={"ID":"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182","Type":"ContainerDied","Data":"4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a"} Nov 01 06:29:07.051834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.051803 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5" event={"ID":"ae3d3a0e-b97e-4cb4-bc04-bc5550c79182","Type":"ContainerDied","Data":"f9bd932819ba695260a64fa81633428acf8bac7a4701a22c75fd68200ed9d231"} Nov 01 06:29:07.051834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.051829 4914 scope.go:122] "RemoveContainer" containerID="4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a" Nov 01 06:29:07.053166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.053144 4914 generic.go:356] "Generic (PLEG): container finished" podID="24379c49-137f-43a6-a40f-d53c32d1ab8d" containerID="57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4" exitCode=0 Nov 01 06:29:07.053257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.053189 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" Nov 01 06:29:07.053257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.053195 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" event={"ID":"24379c49-137f-43a6-a40f-d53c32d1ab8d","Type":"ContainerDied","Data":"57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4"} Nov 01 06:29:07.053257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.053231 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n" event={"ID":"24379c49-137f-43a6-a40f-d53c32d1ab8d","Type":"ContainerDied","Data":"1c7396d67500f0630bb15200763f1dc2381556067718d797d90deff6a553ce4b"} Nov 01 06:29:07.060794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.060778 4914 scope.go:122] "RemoveContainer" containerID="4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a" Nov 01 06:29:07.061082 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:29:07.061059 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a\": container with ID starting with 4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a not found: ID does not exist" containerID="4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a" Nov 01 06:29:07.061189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.061089 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a"} err="failed to get container status \"4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a\": rpc error: code = NotFound desc = could not find container \"4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a\": container with ID starting with 4797d9e5b751e0947ebfdb0fda0702514c9c090e0ba10aadd355e30afa46546a not found: ID does not exist" Nov 01 06:29:07.061189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.061112 4914 scope.go:122] "RemoveContainer" containerID="57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4" Nov 01 06:29:07.069406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.069388 4914 scope.go:122] "RemoveContainer" containerID="57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4" Nov 01 06:29:07.069660 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:29:07.069643 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4\": container with ID starting with 57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4 not found: ID does not exist" containerID="57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4" Nov 01 06:29:07.069748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.069667 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4"} err="failed to get container status \"57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4\": rpc error: code = NotFound desc = could not find container \"57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4\": container with ID starting with 57dc2f57d747283e5fe47a24f755e3123a763e369b0290bdabc179eb7b2e97d4 not found: ID does not exist" Nov 01 06:29:07.123647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.123611 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgvn\" (UniqueName: \"kubernetes.io/projected/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182-kube-api-access-8hgvn\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:07.123647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.123649 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mhs\" (UniqueName: \"kubernetes.io/projected/24379c49-137f-43a6-a40f-d53c32d1ab8d-kube-api-access-t5mhs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:29:07.200203 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.200173 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5"] Nov 01 06:29:07.206062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.206038 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b354e3-cbf5c5747-867n5"] Nov 01 06:29:07.312655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.312595 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n"] Nov 01 06:29:07.316444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:07.316423 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b354e3-554dd8d747-pfv8n"] Nov 01 06:29:07.591411 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:29:07.591347 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:29:08.616450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:08.616413 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24379c49-137f-43a6-a40f-d53c32d1ab8d" path="/var/lib/kubelet/pods/24379c49-137f-43a6-a40f-d53c32d1ab8d/volumes" Nov 01 06:29:08.616948 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:08.616927 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" path="/var/lib/kubelet/pods/ae3d3a0e-b97e-4cb4-bc04-bc5550c79182/volumes" Nov 01 06:29:11.461089 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461058 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7"] Nov 01 06:29:11.461647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461631 4914 cpu_manager.go:414] "containerMap: removing container" podUID="ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" containerName="payload-processing" Nov 01 06:29:11.461733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461648 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" containerName="payload-processing" Nov 01 06:29:11.461733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461678 4914 cpu_manager.go:414] "containerMap: removing container" podUID="24379c49-137f-43a6-a40f-d53c32d1ab8d" containerName="payload-pre-processing" Nov 01 06:29:11.461733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461683 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24379c49-137f-43a6-a40f-d53c32d1ab8d" containerName="payload-pre-processing" Nov 01 06:29:11.461897 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461767 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3d3a0e-b97e-4cb4-bc04-bc5550c79182" containerName="payload-processing" Nov 01 06:29:11.461897 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.461777 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="24379c49-137f-43a6-a40f-d53c32d1ab8d" containerName="payload-pre-processing" Nov 01 06:29:11.466536 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.466515 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.518382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.518363 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-e7b2a3-serving-cert\"" Nov 01 06:29:11.558549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.558525 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.558640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.558587 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdtr7\" (UniqueName: \"kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.558712 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.558691 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.558748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.558740 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.571138 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.571114 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7"] Nov 01 06:29:11.659738 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.659706 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.659888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.659781 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.659888 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:29:11.659820 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-serving-cert: secret "maas-api-e2e-praxis-a-e7b2a3-serving-cert" not found Nov 01 06:29:11.659888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.659844 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.660051 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:29:11.659897 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls podName:b221fcc9-bb3c-4694-ba54-57cb8c9e7047 nodeName:}" failed. No retries permitted until 2026-11-01 06:29:12.159874101 +0000 UTC m=+1167.712663293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls") pod "maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" (UID: "b221fcc9-bb3c-4694-ba54-57cb8c9e7047") : secret "maas-api-e2e-praxis-a-e7b2a3-serving-cert" not found Nov 01 06:29:11.660051 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.659996 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdtr7\" (UniqueName: \"kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.660571 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.660549 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.662490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.662462 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:11.750103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:11.749494 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdtr7\" (UniqueName: \"kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:12.171162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:12.171125 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:12.173402 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:12.173384 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:12.384512 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:12.384485 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:12.752153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:12.752130 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7"] Nov 01 06:29:12.753697 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:29:12.753671 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb221fcc9_bb3c_4694_ba54_57cb8c9e7047.slice/crio-a93df5bcf7adfd47f8b81f0b1e9e3828bf4495f1452ff56301f1de995e236a56 WatchSource:0}: Error finding container a93df5bcf7adfd47f8b81f0b1e9e3828bf4495f1452ff56301f1de995e236a56: Status 404 returned error can't find the container with id a93df5bcf7adfd47f8b81f0b1e9e3828bf4495f1452ff56301f1de995e236a56 Nov 01 06:29:13.091676 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:13.091608 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" event={"ID":"b221fcc9-bb3c-4694-ba54-57cb8c9e7047","Type":"ContainerStarted","Data":"a93df5bcf7adfd47f8b81f0b1e9e3828bf4495f1452ff56301f1de995e236a56"} Nov 01 06:29:14.098448 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:14.098408 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" event={"ID":"b221fcc9-bb3c-4694-ba54-57cb8c9e7047","Type":"ContainerStarted","Data":"f5a8036ddf471c74be9f4aab1fef62a4b9556d7c550be0588fc2263380796b36"} Nov 01 06:29:14.098868 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:14.098471 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:14.258861 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:14.258797 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" podStartSLOduration=2.351987418 podStartE2EDuration="3.258775787s" totalImagesPullingTime="906.788369ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:29:14.245570394 +0000 UTC m=+1169.798359587" watchObservedRunningTime="2026-11-01 06:29:14.258775787 +0000 UTC m=+1169.811564980" Nov 01 06:29:20.132373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:20.132344 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:29:37.591693 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:29:37.591660 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:29:44.744916 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:44.744889 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:29:44.747428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:29:44.747402 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:30:05.058513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:05.058437 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-q2fb6_f72e4bf9-85e3-4aff-b5ff-f98bcbe80124/payload-processing/0.log" Nov 01 06:30:05.295990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:05.295962 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-q2fb6_f72e4bf9-85e3-4aff-b5ff-f98bcbe80124/payload-processing/0.log" Nov 01 06:30:07.636649 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:30:07.636615 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:30:13.131261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.131228 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz"] Nov 01 06:30:13.135544 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.135521 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177539 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177513 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177568 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177586 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177644 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177817 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177761 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177874 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177817 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177874 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177849 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s659r\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177918 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.177970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.177937 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.187317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.187295 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-390a75-openshift-default-dockercfg-klxnc\"" Nov 01 06:30:13.240097 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.240073 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz"] Nov 01 06:30:13.278648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278611 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.278798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278662 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.278798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278742 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.278966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278807 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.278966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278832 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.278966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278871 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.278966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278934 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.278997 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.279020 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s659r\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279293 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.279160 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.279474 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.279497 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279759 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.279735 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.279913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.279817 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.281039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.281018 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.282334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.282311 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.335943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.335920 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s659r\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.336336 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.336319 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token\") pod \"e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.462349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.450538 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:13.918398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.918374 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz"] Nov 01 06:30:13.919585 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:30:13.919564 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ff29ad_3aaa_4333_b412_6a2bce942884.slice/crio-86fe71f280b168010b32f4910f3ba5a0d71333b3bbf9e4a5e253771d71e29115 WatchSource:0}: Error finding container 86fe71f280b168010b32f4910f3ba5a0d71333b3bbf9e4a5e253771d71e29115: Status 404 returned error can't find the container with id 86fe71f280b168010b32f4910f3ba5a0d71333b3bbf9e4a5e253771d71e29115 Nov 01 06:30:13.921895 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.921861 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:30:13.921982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.921925 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:30:13.921982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:13.921952 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:30:14.584695 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:14.584657 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" event={"ID":"15ff29ad-3aaa-4333-b412-6a2bce942884","Type":"ContainerStarted","Data":"e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8"} Nov 01 06:30:14.585178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:14.584718 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" event={"ID":"15ff29ad-3aaa-4333-b412-6a2bce942884","Type":"ContainerStarted","Data":"86fe71f280b168010b32f4910f3ba5a0d71333b3bbf9e4a5e253771d71e29115"} Nov 01 06:30:15.451540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:15.451508 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:15.456496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:15.456472 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:15.572705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:15.572654 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podStartSLOduration=2.572640975 podStartE2EDuration="2.572640975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:30:14.699006411 +0000 UTC m=+1230.251795602" watchObservedRunningTime="2026-11-01 06:30:15.572640975 +0000 UTC m=+1231.125430163" Nov 01 06:30:15.590197 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:15.590167 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:15.591032 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:15.591014 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:30:36.801821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:36.801766 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:30:36.802595 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:36.802546 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:37.592118 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:30:37.592071 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:30:53.499492 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:53.499435 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:30:53.500216 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:53.500183 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:53.962023 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:53.961989 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:30:53.962357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:53.962328 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:30:59.485526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.485491 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw"] Nov 01 06:30:59.489723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.489695 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537300 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537357 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537488 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537607 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537573 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537656 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvr5j\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537722 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537693 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537751 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537792 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.537833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.537821 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.562262 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.562234 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw"] Nov 01 06:30:59.595325 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.595302 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-25719d-openshift-default-dockercfg-hmbnn\"" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650034 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650109 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650149 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650202 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650290 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650359 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650418 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvr5j\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650456 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650490 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.650951 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.651608 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.651841 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.652060 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.661300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.656839 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.779204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.779126 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.787999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.782317 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvr5j\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.793388 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.787923 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.794486 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.794459 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy\") pod \"e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:30:59.805747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:30:59.805513 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:31:00.282060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:00.282033 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw"] Nov 01 06:31:00.282892 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:31:00.282868 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3460fb3_9719_4dc5_9aa9_b7e23230c69a.slice/crio-20b20b1a5a4c758947726a000162eef3babe9edc81b64e7e3afc196589772bbb WatchSource:0}: Error finding container 20b20b1a5a4c758947726a000162eef3babe9edc81b64e7e3afc196589772bbb: Status 404 returned error can't find the container with id 20b20b1a5a4c758947726a000162eef3babe9edc81b64e7e3afc196589772bbb Nov 01 06:31:00.285136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:00.285103 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:31:00.285206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:00.285167 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:31:00.285206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:00.285196 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:31:01.070439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:01.070409 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" event={"ID":"b3460fb3-9719-4dc5-9aa9-b7e23230c69a","Type":"ContainerStarted","Data":"bb0e463cbd8e665b530aab62ed0393f77c27bd30ad2b7324cc1f966015efb3a4"} Nov 01 06:31:01.070903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:01.070465 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" event={"ID":"b3460fb3-9719-4dc5-9aa9-b7e23230c69a","Type":"ContainerStarted","Data":"20b20b1a5a4c758947726a000162eef3babe9edc81b64e7e3afc196589772bbb"} Nov 01 06:31:01.185090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:01.185034 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podStartSLOduration=2.185018174 podStartE2EDuration="2.185018174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:01.184523127 +0000 UTC m=+1276.737312318" watchObservedRunningTime="2026-11-01 06:31:01.185018174 +0000 UTC m=+1276.737807366" Nov 01 06:31:01.806704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:01.806671 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:31:01.811884 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:01.811861 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:31:02.076264 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:02.076185 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:31:02.076949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:02.076933 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:31:05.621188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.621149 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l"] Nov 01 06:31:05.625204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.625181 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.679537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.679512 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-390a75-serving-cert\"" Nov 01 06:31:05.695451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.695421 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.695451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.695456 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.695640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.695475 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszsh\" (UniqueName: \"kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.695640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.695604 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.730793 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.730770 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l"] Nov 01 06:31:05.796760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.796735 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.796880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.796763 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.796880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.796780 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszsh\" (UniqueName: \"kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.796880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.796841 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.797060 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:31:05.797039 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-serving-cert: secret "maas-api-e2e-praxis-b-390a75-serving-cert" not found Nov 01 06:31:05.797152 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:31:05.797139 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls podName:c4f3348f-1031-4609-a8ce-603f99af9b39 nodeName:}" failed. No retries permitted until 2026-11-01 06:31:06.297115111 +0000 UTC m=+1281.849904318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls") pod "maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" (UID: "c4f3348f-1031-4609-a8ce-603f99af9b39") : secret "maas-api-e2e-praxis-b-390a75-serving-cert" not found Nov 01 06:31:05.797526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.797510 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.799223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.799205 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:05.853362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:05.853332 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszsh\" (UniqueName: \"kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:06.306071 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:06.306039 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:06.308517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:06.308485 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\") pod \"maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:06.544604 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:06.544553 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:06.971931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:06.971885 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l"] Nov 01 06:31:06.980629 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:31:06.980596 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f3348f_1031_4609_a8ce_603f99af9b39.slice/crio-8ffa302b30336193e3d76f1f347545739484a0f1e3a2731ad568187c4ce87659 WatchSource:0}: Error finding container 8ffa302b30336193e3d76f1f347545739484a0f1e3a2731ad568187c4ce87659: Status 404 returned error can't find the container with id 8ffa302b30336193e3d76f1f347545739484a0f1e3a2731ad568187c4ce87659 Nov 01 06:31:07.119557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:07.119516 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" event={"ID":"c4f3348f-1031-4609-a8ce-603f99af9b39","Type":"ContainerStarted","Data":"8ffa302b30336193e3d76f1f347545739484a0f1e3a2731ad568187c4ce87659"} Nov 01 06:31:07.636078 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:31:07.636018 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:31:09.157535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:09.157498 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" event={"ID":"c4f3348f-1031-4609-a8ce-603f99af9b39","Type":"ContainerStarted","Data":"628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523"} Nov 01 06:31:09.157908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:09.157568 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:09.272019 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:09.271976 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" podStartSLOduration=3.305184342 podStartE2EDuration="4.271963693s" totalImagesPullingTime="966.779351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:31:09.269388198 +0000 UTC m=+1284.822177387" watchObservedRunningTime="2026-11-01 06:31:09.271963693 +0000 UTC m=+1284.824752928" Nov 01 06:31:15.169120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:15.169094 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:31:27.997015 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:27.996924 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-q2fb6_f72e4bf9-85e3-4aff-b5ff-f98bcbe80124/payload-processing/0.log" Nov 01 06:31:28.259338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:31:28.259259 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-q2fb6_f72e4bf9-85e3-4aff-b5ff-f98bcbe80124/payload-processing/0.log" Nov 01 06:31:37.596467 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:31:37.596426 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:32:07.593254 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:32:07.593214 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:32:16.634921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:16.634416 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:32:16.634921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:16.634484 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:32:18.816889 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:18.816846 4914 generic.go:356] "Generic (PLEG): container finished" podID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerID="e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8" exitCode=0 Nov 01 06:32:18.817359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:18.816915 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" event={"ID":"15ff29ad-3aaa-4333-b412-6a2bce942884","Type":"ContainerDied","Data":"e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8"} Nov 01 06:32:18.817359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:18.817346 4914 scope.go:122] "RemoveContainer" containerID="e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8" Nov 01 06:32:18.818904 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:18.818876 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:32:18.818984 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:18.818938 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:32:18.818984 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:18.818968 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:32:19.823764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:19.823720 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" event={"ID":"15ff29ad-3aaa-4333-b412-6a2bce942884","Type":"ContainerStarted","Data":"9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56"} Nov 01 06:32:20.453300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:20.452657 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:32:20.453300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:20.452782 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:32:20.453300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:20.452832 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:32:21.453710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:21.453672 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:32:21.454330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:21.454294 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:32:22.461665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:22.461628 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:32:22.462244 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:22.462214 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:32:23.459302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:23.453703 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:32:23.459302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:23.453768 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:32:24.454302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:24.452768 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 01 06:32:24.454302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:24.452834 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 01 06:32:25.456017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:25.455980 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:32:25.456445 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:25.456396 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:32:25.457187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:25.457169 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:32:26.875217 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:26.875123 4914 generic.go:356] "Generic (PLEG): container finished" podID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerID="bb0e463cbd8e665b530aab62ed0393f77c27bd30ad2b7324cc1f966015efb3a4" exitCode=0 Nov 01 06:32:26.875725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:26.875212 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" event={"ID":"b3460fb3-9719-4dc5-9aa9-b7e23230c69a","Type":"ContainerDied","Data":"bb0e463cbd8e665b530aab62ed0393f77c27bd30ad2b7324cc1f966015efb3a4"} Nov 01 06:32:26.875772 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:26.875755 4914 scope.go:122] "RemoveContainer" containerID="bb0e463cbd8e665b530aab62ed0393f77c27bd30ad2b7324cc1f966015efb3a4" Nov 01 06:32:26.877119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:26.877092 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:32:26.877185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:26.877162 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:32:26.877220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:26.877193 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:32:27.883747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:27.883714 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" event={"ID":"b3460fb3-9719-4dc5-9aa9-b7e23230c69a","Type":"ContainerStarted","Data":"ac9f56462feb89dea1b1ce302ee914062820df3924324bf151007559c7068acf"} Nov 01 06:32:28.806513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:28.806479 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:32:28.807363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:28.807339 4914 patch_prober.go:29] interesting pod/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 06:32:28.807482 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:28.807396 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 06:32:29.806389 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:29.806353 4914 patch_prober.go:29] interesting pod/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 06:32:29.806783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:29.806422 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 06:32:30.807187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:30.807155 4914 patch_prober.go:29] interesting pod/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 06:32:30.807588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:30.807220 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 06:32:31.806942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:31.806903 4914 patch_prober.go:29] interesting pod/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 06:32:31.807134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:31.806971 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 06:32:32.076343 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:32.076251 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:32:32.806517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:32.806473 4914 patch_prober.go:29] interesting pod/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 01 06:32:32.806739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:32.806540 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 01 06:32:33.810739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:33.810710 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:32:33.811876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:33.811857 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 06:32:37.591500 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:32:37.591461 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:32:48.904258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:48.904222 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4"] Nov 01 06:32:48.904871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:48.904485 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" podUID="826ee346-5b8e-45af-bdc4-15d2d09ff953" containerName="maas-api" containerID="cri-o://d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81" gracePeriod=30 Nov 01 06:32:49.237793 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.237767 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:32:49.296757 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.296727 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca\") pod \"826ee346-5b8e-45af-bdc4-15d2d09ff953\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " Nov 01 06:32:49.296965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.296793 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs\") pod \"826ee346-5b8e-45af-bdc4-15d2d09ff953\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " Nov 01 06:32:49.296965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.296878 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\") pod \"826ee346-5b8e-45af-bdc4-15d2d09ff953\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " Nov 01 06:32:49.296965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.296922 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn\" (UniqueName: \"kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn\") pod \"826ee346-5b8e-45af-bdc4-15d2d09ff953\" (UID: \"826ee346-5b8e-45af-bdc4-15d2d09ff953\") " Nov 01 06:32:49.297171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.297119 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca" pod "826ee346-5b8e-45af-bdc4-15d2d09ff953" (UID: "826ee346-5b8e-45af-bdc4-15d2d09ff953"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:32:49.297405 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.297382 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/826ee346-5b8e-45af-bdc4-15d2d09ff953-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:49.299133 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.299109 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn" pod "826ee346-5b8e-45af-bdc4-15d2d09ff953" (UID: "826ee346-5b8e-45af-bdc4-15d2d09ff953"). InnerVolumeSpecName "kube-api-access-rl6gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:32:49.299216 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.299153 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls" pod "826ee346-5b8e-45af-bdc4-15d2d09ff953" (UID: "826ee346-5b8e-45af-bdc4-15d2d09ff953"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:32:49.299216 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.299162 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs" pod "826ee346-5b8e-45af-bdc4-15d2d09ff953" (UID: "826ee346-5b8e-45af-bdc4-15d2d09ff953"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:32:49.398251 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.398213 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:49.398251 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.398251 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/826ee346-5b8e-45af-bdc4-15d2d09ff953-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:49.398485 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:49.398265 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6gn\" (UniqueName: \"kubernetes.io/projected/826ee346-5b8e-45af-bdc4-15d2d09ff953-kube-api-access-rl6gn\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:32:50.017127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.017094 4914 generic.go:356] "Generic (PLEG): container finished" podID="826ee346-5b8e-45af-bdc4-15d2d09ff953" containerID="d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81" exitCode=0 Nov 01 06:32:50.017610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.017142 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" event={"ID":"826ee346-5b8e-45af-bdc4-15d2d09ff953","Type":"ContainerDied","Data":"d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81"} Nov 01 06:32:50.017610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.017159 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" Nov 01 06:32:50.017610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.017200 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4" event={"ID":"826ee346-5b8e-45af-bdc4-15d2d09ff953","Type":"ContainerDied","Data":"3df483b8da3192828020d0b1762a5abf1abeec385c63b025242683538060be91"} Nov 01 06:32:50.017610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.017226 4914 scope.go:122] "RemoveContainer" containerID="d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81" Nov 01 06:32:50.033791 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.033763 4914 scope.go:122] "RemoveContainer" containerID="d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81" Nov 01 06:32:50.034229 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:32:50.034210 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81\": container with ID starting with d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81 not found: ID does not exist" containerID="d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81" Nov 01 06:32:50.034341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.034236 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81"} err="failed to get container status \"d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81\": rpc error: code = NotFound desc = could not find container \"d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81\": container with ID starting with d9c91a141c4c99b719f2770a197ea98847d5b1e6daf3f861af2de1f3f1e2eb81 not found: ID does not exist" Nov 01 06:32:50.182018 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.181984 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4"] Nov 01 06:32:50.187429 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.187402 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd9d24-65c8579c96-4bsq4"] Nov 01 06:32:50.616597 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:50.616571 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826ee346-5b8e-45af-bdc4-15d2d09ff953" path="/var/lib/kubelet/pods/826ee346-5b8e-45af-bdc4-15d2d09ff953/volumes" Nov 01 06:32:51.231387 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.231355 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn"] Nov 01 06:32:51.232125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.232102 4914 cpu_manager.go:414] "containerMap: removing container" podUID="826ee346-5b8e-45af-bdc4-15d2d09ff953" containerName="maas-api" Nov 01 06:32:51.232223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.232127 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826ee346-5b8e-45af-bdc4-15d2d09ff953" containerName="maas-api" Nov 01 06:32:51.232345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.232331 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="826ee346-5b8e-45af-bdc4-15d2d09ff953" containerName="maas-api" Nov 01 06:32:51.239784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.239746 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.292534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.292503 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn"] Nov 01 06:32:51.326521 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.326491 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.326701 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.326617 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mqc\" (UniqueName: \"kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.326701 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.326644 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.326701 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.326671 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.326841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.326778 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.326841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.326833 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.338541 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.338512 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-e7b2a3\"/\"kube-root-ca.crt\"" Nov 01 06:32:51.338674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.338567 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-e7b2a3\"/\"openshift-service-ca.crt\"" Nov 01 06:32:51.341321 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.341264 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-e7b2a3\"/\"default-dockercfg-qd5vc\"" Nov 01 06:32:51.341321 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.341315 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-e7b2a3\"/\"praxis-route-e7b2a3-kserve-self-signed-certs\"" Nov 01 06:32:51.432191 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.432154 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.434778 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.434749 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mqc\" (UniqueName: \"kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.434980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.434965 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.435089 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.435076 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.435383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.435347 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.435620 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.435599 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.437665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.434584 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.438678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.438402 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.438808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.438620 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.452168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.452140 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.470956 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.470917 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.503253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.503182 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mqc\" (UniqueName: \"kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc\") pod \"praxis-route-e7b2a3-kserve-69d8cf6469-2llqn\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.553605 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.553569 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:32:51.926403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:51.926375 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn"] Nov 01 06:32:51.927229 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:32:51.927187 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3928a9cd_66b7_4f47_bbb0_105b4e22ab44.slice/crio-2bf39b771675cfd9c404a5605461e8ea8df113e23dbae768fe6c6f1c26cdd6ce WatchSource:0}: Error finding container 2bf39b771675cfd9c404a5605461e8ea8df113e23dbae768fe6c6f1c26cdd6ce: Status 404 returned error can't find the container with id 2bf39b771675cfd9c404a5605461e8ea8df113e23dbae768fe6c6f1c26cdd6ce Nov 01 06:32:52.031317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:52.031264 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" event={"ID":"3928a9cd-66b7-4f47-bbb0-105b4e22ab44","Type":"ContainerStarted","Data":"6bc5e69b785b0ee657fcc7edbfa5016068e7e93448cfd793a4086f8e6ab93c34"} Nov 01 06:32:52.031484 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:52.031327 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" event={"ID":"3928a9cd-66b7-4f47-bbb0-105b4e22ab44","Type":"ContainerStarted","Data":"2bf39b771675cfd9c404a5605461e8ea8df113e23dbae768fe6c6f1c26cdd6ce"} Nov 01 06:32:52.161801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:52.161749 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" podStartSLOduration=1.16173547 podStartE2EDuration="1.16173547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:32:52.160976508 +0000 UTC m=+1387.713765701" watchObservedRunningTime="2026-11-01 06:32:52.16173547 +0000 UTC m=+1387.714524663" Nov 01 06:32:52.554688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:32:52.554658 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:33:01.554491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:01.554409 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:33:01.567071 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:01.567040 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:33:02.118393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:02.118366 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:33:07.593292 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:07.593236 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:33:10.451555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.451521 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh"] Nov 01 06:33:10.455729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.455709 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.508595 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.508556 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-56ae52d5-serving-cert\"" Nov 01 06:33:10.536534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.536506 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.536707 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.536545 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.536707 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.536669 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.536707 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.536698 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8cr\" (UniqueName: \"kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.561736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.561705 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh"] Nov 01 06:33:10.637939 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.637906 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.638140 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.637957 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8cr\" (UniqueName: \"kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.638343 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.638243 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.638708 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.638422 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.638708 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:10.638524 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-serving-cert: secret "maas-api-e2e-ait-56ae52d5-serving-cert" not found Nov 01 06:33:10.638708 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:10.638592 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls podName:38313716-cb5a-4fce-afc3-56c2f84de480 nodeName:}" failed. No retries permitted until 2026-11-01 06:33:11.138572178 +0000 UTC m=+1406.691361352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls") pod "maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" (UID: "38313716-cb5a-4fce-afc3-56c2f84de480") : secret "maas-api-e2e-ait-56ae52d5-serving-cert" not found Nov 01 06:33:10.639019 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.638998 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.640751 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.640730 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:10.695945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:10.695907 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8cr\" (UniqueName: \"kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:11.172576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.172532 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:11.175021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.174998 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\") pod \"maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:11.370608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.370575 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:11.865911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.865874 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl"] Nov 01 06:33:11.871301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.871253 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh"] Nov 01 06:33:11.871425 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.871368 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:11.902377 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:33:11.902349 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38313716_cb5a_4fce_afc3_56c2f84de480.slice/crio-ba6e1881d7fbc4796112ceeb98086cc3380f3b2e76b6183ec7347e08daab22cc WatchSource:0}: Error finding container ba6e1881d7fbc4796112ceeb98086cc3380f3b2e76b6183ec7347e08daab22cc: Status 404 returned error can't find the container with id ba6e1881d7fbc4796112ceeb98086cc3380f3b2e76b6183ec7347e08daab22cc Nov 01 06:33:11.903783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.903768 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:33:11.925130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.925106 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl"] Nov 01 06:33:11.931696 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.931677 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1ff1dc\"" Nov 01 06:33:11.955700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:11.955666 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1ff1dc-dockercfg-4d2k7\"" Nov 01 06:33:12.016414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.016379 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xh5\" (UniqueName: \"kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5\") pod \"payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.016573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.016534 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.117576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.117489 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xh5\" (UniqueName: \"kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5\") pod \"payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.117744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.117580 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.118368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.118344 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.165924 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.165893 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" event={"ID":"38313716-cb5a-4fce-afc3-56c2f84de480","Type":"ContainerStarted","Data":"ba6e1881d7fbc4796112ceeb98086cc3380f3b2e76b6183ec7347e08daab22cc"} Nov 01 06:33:12.174589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.174560 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xh5\" (UniqueName: \"kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5\") pod \"payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.185129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.185109 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:12.557016 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:33:12.556985 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489380c_db6b_4d39_aacc_ceaf37ab5c81.slice/crio-250bd354b2e986e550f24db1d8f1c23018c292e3031067f41a3107841b60a133 WatchSource:0}: Error finding container 250bd354b2e986e550f24db1d8f1c23018c292e3031067f41a3107841b60a133: Status 404 returned error can't find the container with id 250bd354b2e986e550f24db1d8f1c23018c292e3031067f41a3107841b60a133 Nov 01 06:33:12.557410 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:12.557386 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl"] Nov 01 06:33:13.174047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.174008 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" event={"ID":"a489380c-db6b-4d39-aacc-ceaf37ab5c81","Type":"ContainerStarted","Data":"f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b"} Nov 01 06:33:13.174527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.174071 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" event={"ID":"a489380c-db6b-4d39-aacc-ceaf37ab5c81","Type":"ContainerStarted","Data":"250bd354b2e986e550f24db1d8f1c23018c292e3031067f41a3107841b60a133"} Nov 01 06:33:13.174527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.174100 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:13.175682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.175662 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" event={"ID":"38313716-cb5a-4fce-afc3-56c2f84de480","Type":"ContainerStarted","Data":"055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444"} Nov 01 06:33:13.175740 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.175705 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:13.297931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.297483 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" podStartSLOduration=2.297456051 podStartE2EDuration="2.297456051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:13.291099029 +0000 UTC m=+1408.843888222" watchObservedRunningTime="2026-11-01 06:33:13.297456051 +0000 UTC m=+1408.850245271" Nov 01 06:33:13.407222 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:13.407156 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" podStartSLOduration=2.484313362 podStartE2EDuration="3.407142057s" totalImagesPullingTime="922.828695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:33:13.405038983 +0000 UTC m=+1408.957828175" watchObservedRunningTime="2026-11-01 06:33:13.407142057 +0000 UTC m=+1408.959931248" Nov 01 06:33:15.030143 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.030110 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr"] Nov 01 06:33:15.030600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.030351 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" podUID="c50a8686-d2a1-4302-bdac-61715a6e77d8" containerName="maas-api" containerID="cri-o://b8fa2a0a0aa172e4b5637be28ba7fc196f11bd4a8109d94f8097e493358169cc" gracePeriod=30 Nov 01 06:33:15.190175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.190141 4914 generic.go:356] "Generic (PLEG): container finished" podID="c50a8686-d2a1-4302-bdac-61715a6e77d8" containerID="b8fa2a0a0aa172e4b5637be28ba7fc196f11bd4a8109d94f8097e493358169cc" exitCode=0 Nov 01 06:33:15.190389 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.190220 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" event={"ID":"c50a8686-d2a1-4302-bdac-61715a6e77d8","Type":"ContainerDied","Data":"b8fa2a0a0aa172e4b5637be28ba7fc196f11bd4a8109d94f8097e493358169cc"} Nov 01 06:33:15.347458 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.347431 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:33:15.503982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.503949 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\") pod \"c50a8686-d2a1-4302-bdac-61715a6e77d8\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " Nov 01 06:33:15.504185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.504017 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z\" (UniqueName: \"kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z\") pod \"c50a8686-d2a1-4302-bdac-61715a6e77d8\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " Nov 01 06:33:15.504253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.504218 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca\") pod \"c50a8686-d2a1-4302-bdac-61715a6e77d8\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " Nov 01 06:33:15.504355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.504254 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs\") pod \"c50a8686-d2a1-4302-bdac-61715a6e77d8\" (UID: \"c50a8686-d2a1-4302-bdac-61715a6e77d8\") " Nov 01 06:33:15.504916 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.504602 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca" pod "c50a8686-d2a1-4302-bdac-61715a6e77d8" (UID: "c50a8686-d2a1-4302-bdac-61715a6e77d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:15.504916 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.504851 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50a8686-d2a1-4302-bdac-61715a6e77d8-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:15.506758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.506734 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls" pod "c50a8686-d2a1-4302-bdac-61715a6e77d8" (UID: "c50a8686-d2a1-4302-bdac-61715a6e77d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:15.506907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.506863 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z" pod "c50a8686-d2a1-4302-bdac-61715a6e77d8" (UID: "c50a8686-d2a1-4302-bdac-61715a6e77d8"). InnerVolumeSpecName "kube-api-access-6q78z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:15.507155 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.507139 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs" pod "c50a8686-d2a1-4302-bdac-61715a6e77d8" (UID: "c50a8686-d2a1-4302-bdac-61715a6e77d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:33:15.606498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.606470 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:15.606498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.606497 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50a8686-d2a1-4302-bdac-61715a6e77d8-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:15.606668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:15.606507 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q78z\" (UniqueName: \"kubernetes.io/projected/c50a8686-d2a1-4302-bdac-61715a6e77d8-kube-api-access-6q78z\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:16.202870 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:16.202839 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" Nov 01 06:33:16.203308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:16.202859 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr" event={"ID":"c50a8686-d2a1-4302-bdac-61715a6e77d8","Type":"ContainerDied","Data":"cff418b3967510da93ed8eb70a5d0034f4397a4a4252140eb8688291f804fa12"} Nov 01 06:33:16.203308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:16.202955 4914 scope.go:122] "RemoveContainer" containerID="b8fa2a0a0aa172e4b5637be28ba7fc196f11bd4a8109d94f8097e493358169cc" Nov 01 06:33:16.367858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:16.367824 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr"] Nov 01 06:33:16.417798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:16.417767 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b354e3-848459b796-tcmpr"] Nov 01 06:33:16.617490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:16.617459 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50a8686-d2a1-4302-bdac-61715a6e77d8" path="/var/lib/kubelet/pods/c50a8686-d2a1-4302-bdac-61715a6e77d8/volumes" Nov 01 06:33:19.187054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:19.187024 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:33:24.187776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:24.185822 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 06:33:24.293204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:24.293169 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9"] Nov 01 06:33:24.293464 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:24.293437 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" containerID="cri-o://366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1" gracePeriod=30 Nov 01 06:33:29.560488 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.560428 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:33:29.725123 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725072 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.725476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725160 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.725476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725241 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.725476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725312 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.725476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725339 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.725476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725388 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.725476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725456 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.726198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725583 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.726198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725631 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket\") pod \"bd0265ad-9751-4043-915a-7ccfe6cec445\" (UID: \"bd0265ad-9751-4043-915a-7ccfe6cec445\") " Nov 01 06:33:29.726198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725587 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:29.726198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.725989 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:29.726724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.726666 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:29.726998 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.726972 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.727208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.727152 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0265ad-9751-4043-915a-7ccfe6cec445-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.727329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.727296 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:29.727963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.727926 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:29.728211 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.728187 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:29.729889 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.729823 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:33:29.729987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.729918 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:29.730313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.730270 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp" pod "bd0265ad-9751-4043-915a-7ccfe6cec445" (UID: "bd0265ad-9751-4043-915a-7ccfe6cec445"). InnerVolumeSpecName "kube-api-access-gzqcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.827980 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqcp\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-kube-api-access-gzqcp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.828015 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.828024 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.828033 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.828043 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.828051 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:29.828063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:29.828061 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0265ad-9751-4043-915a-7ccfe6cec445-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:30.295552 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.295519 4914 generic.go:356] "Generic (PLEG): container finished" podID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerID="366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1" exitCode=0 Nov 01 06:33:30.295732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.295580 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" Nov 01 06:33:30.295732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.295607 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" event={"ID":"bd0265ad-9751-4043-915a-7ccfe6cec445","Type":"ContainerDied","Data":"366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1"} Nov 01 06:33:30.295732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.295668 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9" event={"ID":"bd0265ad-9751-4043-915a-7ccfe6cec445","Type":"ContainerDied","Data":"98dcc5430d6ea08485fb81dd15d1ec1325e18b86e1a840ed37d3e3feaa57a7b3"} Nov 01 06:33:30.295732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.295699 4914 scope.go:122] "RemoveContainer" containerID="366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1" Nov 01 06:33:30.306646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.306625 4914 scope.go:122] "RemoveContainer" containerID="32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b" Nov 01 06:33:30.316024 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.316004 4914 scope.go:122] "RemoveContainer" containerID="366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1" Nov 01 06:33:30.316299 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:30.316263 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1\": container with ID starting with 366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1 not found: ID does not exist" containerID="366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1" Nov 01 06:33:30.316413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.316309 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1"} err="failed to get container status \"366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1\": rpc error: code = NotFound desc = could not find container \"366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1\": container with ID starting with 366e7242e30a96f45bd379e7b0fbf93a9a2900f82b38275d9f57e14f65567bc1 not found: ID does not exist" Nov 01 06:33:30.316413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.316334 4914 scope.go:122] "RemoveContainer" containerID="32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b" Nov 01 06:33:30.316610 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:30.316592 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b\": container with ID starting with 32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b not found: ID does not exist" containerID="32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b" Nov 01 06:33:30.316660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.316617 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b"} err="failed to get container status \"32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b\": rpc error: code = NotFound desc = could not find container \"32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b\": container with ID starting with 32709484b9ca051fe35a7a8baf5c7ea7a7ec94ba98dc77965f8e4e65f3f81b3b not found: ID does not exist" Nov 01 06:33:30.510812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.510776 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9"] Nov 01 06:33:30.561007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.560937 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b354e3-openshift-default-fdb8d8f45-ntsn9"] Nov 01 06:33:30.618758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:30.618723 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" path="/var/lib/kubelet/pods/bd0265ad-9751-4043-915a-7ccfe6cec445/volumes" Nov 01 06:33:35.146871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:35.146841 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh"] Nov 01 06:33:35.147340 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:35.147028 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" containerID="cri-o://c3d70008b1e5f649597d686c51bcd072f84284ea3764c43e148ddcc70498d3d9" gracePeriod=30 Nov 01 06:33:37.592779 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:37.592744 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:33:40.359166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.359135 4914 generic.go:356] "Generic (PLEG): container finished" podID="7a8e3672-8cc0-478a-835e-adce15d48683" containerID="c3d70008b1e5f649597d686c51bcd072f84284ea3764c43e148ddcc70498d3d9" exitCode=0 Nov 01 06:33:40.359547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.359220 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" event={"ID":"7a8e3672-8cc0-478a-835e-adce15d48683","Type":"ContainerDied","Data":"c3d70008b1e5f649597d686c51bcd072f84284ea3764c43e148ddcc70498d3d9"} Nov 01 06:33:40.420369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.420303 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:33:40.459836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.459798 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.459887 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.459938 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.459977 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460079 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460116 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:40.460180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460128 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460366 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460252 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460372 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460478 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460391 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:40.460478 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460457 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token\") pod \"7a8e3672-8cc0-478a-835e-adce15d48683\" (UID: \"7a8e3672-8cc0-478a-835e-adce15d48683\") " Nov 01 06:33:40.460622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460596 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:33:40.460885 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460620 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:40.460973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460955 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.461013 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.460982 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.461147 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.461119 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:40.462404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.462377 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:33:40.462514 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.462444 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "kube-api-access-mswgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:40.462768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.462752 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:33:40.463102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.463089 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy" pod "7a8e3672-8cc0-478a-835e-adce15d48683" (UID: "7a8e3672-8cc0-478a-835e-adce15d48683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:33:40.561883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561849 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8e3672-8cc0-478a-835e-adce15d48683-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.561883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561878 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mswgg\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-kube-api-access-mswgg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.561883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561888 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.562108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561897 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8e3672-8cc0-478a-835e-adce15d48683-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.562108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561907 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.562108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561915 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8e3672-8cc0-478a-835e-adce15d48683-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:40.562108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:40.561923 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8e3672-8cc0-478a-835e-adce15d48683-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:33:41.370421 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:41.370387 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" Nov 01 06:33:41.370905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:41.370380 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh" event={"ID":"7a8e3672-8cc0-478a-835e-adce15d48683","Type":"ContainerDied","Data":"c4b84a5e00cce31298a18fd78c006446738e0affb0988aa8d24651567bac29de"} Nov 01 06:33:41.370905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:41.370550 4914 scope.go:122] "RemoveContainer" containerID="c3d70008b1e5f649597d686c51bcd072f84284ea3764c43e148ddcc70498d3d9" Nov 01 06:33:41.587697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:41.587658 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh"] Nov 01 06:33:41.636136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:41.636053 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-x4lgh"] Nov 01 06:33:42.616768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:42.616733 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" path="/var/lib/kubelet/pods/7a8e3672-8cc0-478a-835e-adce15d48683/volumes" Nov 01 06:33:59.366932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.366889 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn"] Nov 01 06:33:59.367732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367709 4914 cpu_manager.go:414] "containerMap: removing container" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" Nov 01 06:33:59.367831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367735 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" Nov 01 06:33:59.367831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367762 4914 cpu_manager.go:414] "containerMap: removing container" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" Nov 01 06:33:59.367831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367771 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" Nov 01 06:33:59.367831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367798 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c50a8686-d2a1-4302-bdac-61715a6e77d8" containerName="maas-api" Nov 01 06:33:59.367831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367805 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50a8686-d2a1-4302-bdac-61715a6e77d8" containerName="maas-api" Nov 01 06:33:59.368064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367981 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" Nov 01 06:33:59.368064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.367998 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8e3672-8cc0-478a-835e-adce15d48683" containerName="istio-proxy" Nov 01 06:33:59.368064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.368011 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50a8686-d2a1-4302-bdac-61715a6e77d8" containerName="maas-api" Nov 01 06:33:59.373002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.372978 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.428721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.428685 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn"] Nov 01 06:33:59.475399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.475371 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1ff1dc-serving-cert\"" Nov 01 06:33:59.492068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.492008 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.492068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.492072 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.492370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.492153 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.492370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.492309 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t959s\" (UniqueName: \"kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.593496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.593458 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.593700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.593506 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.593700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.593530 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.593700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.593580 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t959s\" (UniqueName: \"kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.593700 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:59.593673 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-serving-cert: secret "maas-api-e2e-extmod-b-1ff1dc-serving-cert" not found Nov 01 06:33:59.593920 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:33:59.593750 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls podName:8799c852-3724-4bb3-8576-9fb31e9638af nodeName:}" failed. No retries permitted until 2026-11-01 06:34:00.093728497 +0000 UTC m=+1455.646517688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls") pod "maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" (UID: "8799c852-3724-4bb3-8576-9fb31e9638af") : secret "maas-api-e2e-extmod-b-1ff1dc-serving-cert" not found Nov 01 06:33:59.594123 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.594103 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.596146 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.596125 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:33:59.653230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:33:59.653141 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t959s\" (UniqueName: \"kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:34:00.115576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:00.115533 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:34:00.118173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:00.118138 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:34:00.286537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:00.286494 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:34:00.664504 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:00.664457 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn"] Nov 01 06:34:00.664879 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:34:00.664821 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8799c852_3724_4bb3_8576_9fb31e9638af.slice/crio-2fe78ae621d8c85286abaf75c836a42dea601662caad157c3e2dbdb59c19b4af WatchSource:0}: Error finding container 2fe78ae621d8c85286abaf75c836a42dea601662caad157c3e2dbdb59c19b4af: Status 404 returned error can't find the container with id 2fe78ae621d8c85286abaf75c836a42dea601662caad157c3e2dbdb59c19b4af Nov 01 06:34:01.488397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:01.488362 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" event={"ID":"8799c852-3724-4bb3-8576-9fb31e9638af","Type":"ContainerStarted","Data":"2fe78ae621d8c85286abaf75c836a42dea601662caad157c3e2dbdb59c19b4af"} Nov 01 06:34:02.495354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:02.495320 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" event={"ID":"8799c852-3724-4bb3-8576-9fb31e9638af","Type":"ContainerStarted","Data":"f9a58810da998dc4e4bd021887e0a10d56f4daf8610e1ee3d1f256de3d850576"} Nov 01 06:34:02.495735 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:02.495462 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:34:02.615598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:02.615537 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" podStartSLOduration=2.710355228 podStartE2EDuration="3.615519817s" totalImagesPullingTime="905.164589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:02.613376813 +0000 UTC m=+1458.166166007" watchObservedRunningTime="2026-11-01 06:34:02.615519817 +0000 UTC m=+1458.168309009" Nov 01 06:34:05.403587 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.403548 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj"] Nov 01 06:34:05.404133 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.404119 4914 cpu_manager.go:414] "containerMap: removing container" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" Nov 01 06:34:05.404176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.404134 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" Nov 01 06:34:05.404248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.404239 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0265ad-9751-4043-915a-7ccfe6cec445" containerName="istio-proxy" Nov 01 06:34:05.407784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.407757 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.590799 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.590761 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591001 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.590821 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591001 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.590910 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdms\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591001 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.590954 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.591049 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.591087 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.591105 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.591203 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.591386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.591222 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.612845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.612800 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj"] Nov 01 06:34:05.692917 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.692814 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.692917 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.692882 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.692917 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.692898 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693209 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.692958 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693209 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693109 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693218 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693246 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693263 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693314 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693332 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdms\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693344 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693402 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.693663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.693550 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.694081 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.694007 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.695704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.695678 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.695785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.695724 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.750805 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.750771 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:05.750994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:05.750933 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdms\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms\") pod \"maas-default-gateway-openshift-default-7d9897b54c-476fj\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:06.022394 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:06.022304 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:06.448118 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:06.448090 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj"] Nov 01 06:34:06.448510 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:34:06.448488 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304cf2e2_d0ab_4b7e_aabc_0a105ce15233.slice/crio-fb7567301b1625d7c0625a4f353da5239c54da79011f3aed223b5cf712182ad2 WatchSource:0}: Error finding container fb7567301b1625d7c0625a4f353da5239c54da79011f3aed223b5cf712182ad2: Status 404 returned error can't find the container with id fb7567301b1625d7c0625a4f353da5239c54da79011f3aed223b5cf712182ad2 Nov 01 06:34:06.450965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:06.450935 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:34:06.451033 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:06.450999 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:34:06.451033 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:06.451027 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:34:06.519055 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:06.519029 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" event={"ID":"304cf2e2-d0ab-4b7e-aabc-0a105ce15233","Type":"ContainerStarted","Data":"fb7567301b1625d7c0625a4f353da5239c54da79011f3aed223b5cf712182ad2"} Nov 01 06:34:07.526721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:07.526685 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" event={"ID":"304cf2e2-d0ab-4b7e-aabc-0a105ce15233","Type":"ContainerStarted","Data":"918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78"} Nov 01 06:34:07.596700 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:34:07.596664 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:34:07.743037 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:07.742980 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podStartSLOduration=2.742964776 podStartE2EDuration="2.742964776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:07.692290464 +0000 UTC m=+1463.245079653" watchObservedRunningTime="2026-11-01 06:34:07.742964776 +0000 UTC m=+1463.295753967" Nov 01 06:34:08.022796 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:08.022758 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:08.024474 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:08.024450 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-476fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:34:08.024529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:08.024505 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:34:08.506397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:08.506363 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 06:34:09.023485 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:09.023447 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-476fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:34:09.023873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:09.023502 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:34:09.475729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:09.475686 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn"] Nov 01 06:34:10.023442 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:10.023401 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-476fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:34:10.023643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:10.023462 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:34:11.022972 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:11.022930 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-476fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:34:11.023176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:11.022987 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:34:12.023624 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:12.023590 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-476fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:34:12.023982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:12.023657 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:34:13.022623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:13.022587 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-476fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 01 06:34:13.022797 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:13.022645 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 01 06:34:14.677630 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:14.677461 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:14.678266 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:14.678245 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:14.679313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:14.679259 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:34:24.497171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.497110 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" podUID="3928a9cd-66b7-4f47-bbb0-105b4e22ab44" containerName="main" containerID="cri-o://6bc5e69b785b0ee657fcc7edbfa5016068e7e93448cfd793a4086f8e6ab93c34" gracePeriod=45 Nov 01 06:34:24.740392 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.740358 4914 generic.go:356] "Generic (PLEG): container finished" podID="3928a9cd-66b7-4f47-bbb0-105b4e22ab44" containerID="6bc5e69b785b0ee657fcc7edbfa5016068e7e93448cfd793a4086f8e6ab93c34" exitCode=0 Nov 01 06:34:24.740544 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.740429 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" event={"ID":"3928a9cd-66b7-4f47-bbb0-105b4e22ab44","Type":"ContainerDied","Data":"6bc5e69b785b0ee657fcc7edbfa5016068e7e93448cfd793a4086f8e6ab93c34"} Nov 01 06:34:24.766176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.766149 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.872366 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir\") pod \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.872765 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir" pod "3928a9cd-66b7-4f47-bbb0-105b4e22ab44" (UID: "3928a9cd-66b7-4f47-bbb0-105b4e22ab44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.872905 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs\") pod \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.873451 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache\") pod \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.873504 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc\" (UniqueName: \"kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc\") pod \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.873549 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm\") pod \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.873631 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home\") pod \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\" (UID: \"3928a9cd-66b7-4f47-bbb0-105b4e22ab44\") " Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.874441 4914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tmp-dir\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.874734 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home" pod "3928a9cd-66b7-4f47-bbb0-105b4e22ab44" (UID: "3928a9cd-66b7-4f47-bbb0-105b4e22ab44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:34:24.881298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.875961 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache" pod "3928a9cd-66b7-4f47-bbb0-105b4e22ab44" (UID: "3928a9cd-66b7-4f47-bbb0-105b4e22ab44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:34:24.914535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.914472 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc" pod "3928a9cd-66b7-4f47-bbb0-105b4e22ab44" (UID: "3928a9cd-66b7-4f47-bbb0-105b4e22ab44"). InnerVolumeSpecName "kube-api-access-z5mqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:24.915177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.915129 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm" pod "3928a9cd-66b7-4f47-bbb0-105b4e22ab44" (UID: "3928a9cd-66b7-4f47-bbb0-105b4e22ab44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:34:24.935966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.935579 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs" pod "3928a9cd-66b7-4f47-bbb0-105b4e22ab44" (UID: "3928a9cd-66b7-4f47-bbb0-105b4e22ab44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:24.975903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.975805 4914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-tls-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:24.975903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.975835 4914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-model-cache\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:24.975903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.975851 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mqc\" (UniqueName: \"kubernetes.io/projected/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-kube-api-access-z5mqc\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:24.975903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.975867 4914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-dshm\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:24.975903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:24.975880 4914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3928a9cd-66b7-4f47-bbb0-105b4e22ab44-home\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:25.748580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:25.748541 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" event={"ID":"3928a9cd-66b7-4f47-bbb0-105b4e22ab44","Type":"ContainerDied","Data":"2bf39b771675cfd9c404a5605461e8ea8df113e23dbae768fe6c6f1c26cdd6ce"} Nov 01 06:34:25.749016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:25.748594 4914 scope.go:122] "RemoveContainer" containerID="6bc5e69b785b0ee657fcc7edbfa5016068e7e93448cfd793a4086f8e6ab93c34" Nov 01 06:34:25.749016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:25.748564 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn" Nov 01 06:34:25.913944 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:25.913867 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn"] Nov 01 06:34:25.963931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:25.963904 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-e7b2a3/praxis-route-e7b2a3-kserve-69d8cf6469-2llqn"] Nov 01 06:34:26.616351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:26.616323 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3928a9cd-66b7-4f47-bbb0-105b4e22ab44" path="/var/lib/kubelet/pods/3928a9cd-66b7-4f47-bbb0-105b4e22ab44/volumes" Nov 01 06:34:37.592771 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:34:37.592733 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:34:44.781900 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:44.781867 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:34:44.786627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:44.786603 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:34:46.014956 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:46.014918 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk"] Nov 01 06:34:50.734914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.734880 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn"] Nov 01 06:34:50.735673 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.735650 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3928a9cd-66b7-4f47-bbb0-105b4e22ab44" containerName="main" Nov 01 06:34:50.735673 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.735673 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3928a9cd-66b7-4f47-bbb0-105b4e22ab44" containerName="main" Nov 01 06:34:50.735850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.735836 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3928a9cd-66b7-4f47-bbb0-105b4e22ab44" containerName="main" Nov 01 06:34:50.740641 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.740620 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.846224 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.846192 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn"] Nov 01 06:34:50.915752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.915713 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.915915 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.915778 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.915915 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.915860 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.915995 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.915963 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.915995 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.915989 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.916081 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.916012 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.916081 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.916031 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.916180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.916158 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnn2s\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:50.916256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:50.916242 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.015465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.015376 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerName="payload-processing" containerID="cri-o://fdf926a005a1faf40aaca9e206b2bb2fbf03d5e354150f7f32dd5dc93bb40624" gracePeriod=25 Nov 01 06:34:51.017346 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017322 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnn2s\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017416 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017513 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017572 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017649 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017622 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017698 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017727 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017753 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.017921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017776 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.018029 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017973 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.018029 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.017994 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.018202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.018147 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.018329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.018310 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.018965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.018944 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.020030 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.020006 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.020222 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.020203 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.074003 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.073965 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnn2s\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.074115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.073973 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-r9phn\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.354726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.354696 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:51.779177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.779088 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn"] Nov 01 06:34:51.826363 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:34:51.826327 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f48f22_54e0_4d28_a631_d5e7aef264dd.slice/crio-bc8ced5af420226690c80d4844e5c9cc97a4eb3066ce25e9bb55517f3aa80372 WatchSource:0}: Error finding container bc8ced5af420226690c80d4844e5c9cc97a4eb3066ce25e9bb55517f3aa80372: Status 404 returned error can't find the container with id bc8ced5af420226690c80d4844e5c9cc97a4eb3066ce25e9bb55517f3aa80372 Nov 01 06:34:51.828648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.828614 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:34:51.828711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.828682 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:34:51.828745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.828712 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:34:51.910607 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:51.910579 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" event={"ID":"a2f48f22-54e0-4d28-a631-d5e7aef264dd","Type":"ContainerStarted","Data":"bc8ced5af420226690c80d4844e5c9cc97a4eb3066ce25e9bb55517f3aa80372"} Nov 01 06:34:52.929184 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:52.929143 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" event={"ID":"a2f48f22-54e0-4d28-a631-d5e7aef264dd","Type":"ContainerStarted","Data":"c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4"} Nov 01 06:34:53.098427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:53.098363 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podStartSLOduration=3.098347558 podStartE2EDuration="3.098347558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:34:53.09706235 +0000 UTC m=+1508.649851543" watchObservedRunningTime="2026-11-01 06:34:53.098347558 +0000 UTC m=+1508.651136749" Nov 01 06:34:53.365716 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:53.365679 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:34:53.370477 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:53.370428 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:53.370697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:53.370496 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:34:54.364302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:54.363612 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:54.364302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:54.363686 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:34:55.370302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:55.366642 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:55.370302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:55.366727 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:34:56.364302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:56.360701 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:56.364302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:56.360769 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:34:56.682495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:56.682407 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:34:57.031112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.031027 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh"] Nov 01 06:34:57.031660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.031543 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" podUID="38313716-cb5a-4fce-afc3-56c2f84de480" containerName="maas-api" containerID="cri-o://055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444" gracePeriod=30 Nov 01 06:34:57.355667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.355617 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:57.355866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.355683 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:34:57.622027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.621962 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:34:57.814979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.814938 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\") pod \"38313716-cb5a-4fce-afc3-56c2f84de480\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " Nov 01 06:34:57.815391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.815019 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr\" (UniqueName: \"kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr\") pod \"38313716-cb5a-4fce-afc3-56c2f84de480\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " Nov 01 06:34:57.815391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.815052 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca\") pod \"38313716-cb5a-4fce-afc3-56c2f84de480\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " Nov 01 06:34:57.815391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.815086 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs\") pod \"38313716-cb5a-4fce-afc3-56c2f84de480\" (UID: \"38313716-cb5a-4fce-afc3-56c2f84de480\") " Nov 01 06:34:57.815717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.815695 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca" pod "38313716-cb5a-4fce-afc3-56c2f84de480" (UID: "38313716-cb5a-4fce-afc3-56c2f84de480"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:34:57.817656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.817624 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr" pod "38313716-cb5a-4fce-afc3-56c2f84de480" (UID: "38313716-cb5a-4fce-afc3-56c2f84de480"). InnerVolumeSpecName "kube-api-access-8r8cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:34:57.817790 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.817774 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs" pod "38313716-cb5a-4fce-afc3-56c2f84de480" (UID: "38313716-cb5a-4fce-afc3-56c2f84de480"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:57.817933 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.817914 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls" pod "38313716-cb5a-4fce-afc3-56c2f84de480" (UID: "38313716-cb5a-4fce-afc3-56c2f84de480"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:34:57.916269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.916179 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:57.916269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.916211 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8cr\" (UniqueName: \"kubernetes.io/projected/38313716-cb5a-4fce-afc3-56c2f84de480-kube-api-access-8r8cr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:57.916269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.916221 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38313716-cb5a-4fce-afc3-56c2f84de480-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:57.916269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:57.916229 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38313716-cb5a-4fce-afc3-56c2f84de480-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:34:58.064483 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.064444 4914 generic.go:356] "Generic (PLEG): container finished" podID="38313716-cb5a-4fce-afc3-56c2f84de480" containerID="055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444" exitCode=0 Nov 01 06:34:58.064886 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.064857 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" event={"ID":"38313716-cb5a-4fce-afc3-56c2f84de480","Type":"ContainerDied","Data":"055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444"} Nov 01 06:34:58.065082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.065062 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" event={"ID":"38313716-cb5a-4fce-afc3-56c2f84de480","Type":"ContainerDied","Data":"ba6e1881d7fbc4796112ceeb98086cc3380f3b2e76b6183ec7347e08daab22cc"} Nov 01 06:34:58.065203 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.065190 4914 scope.go:122] "RemoveContainer" containerID="055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444" Nov 01 06:34:58.065529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.065509 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh" Nov 01 06:34:58.111756 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.111725 4914 scope.go:122] "RemoveContainer" containerID="055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444" Nov 01 06:34:58.112755 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:34:58.112306 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444\": container with ID starting with 055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444 not found: ID does not exist" containerID="055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444" Nov 01 06:34:58.112755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.112349 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444"} err="failed to get container status \"055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444\": rpc error: code = NotFound desc = could not find container \"055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444\": container with ID starting with 055ec4e44ef63192069428875f0ef7d3365e29875bd059e60741b0d951da3444 not found: ID does not exist" Nov 01 06:34:58.274702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.274615 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh"] Nov 01 06:34:58.325310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.325261 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ae52d5-7cf7587df7-gnzbh"] Nov 01 06:34:58.355647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.355605 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:58.355801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.355662 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:34:58.617329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:58.617295 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38313716-cb5a-4fce-afc3-56c2f84de480" path="/var/lib/kubelet/pods/38313716-cb5a-4fce-afc3-56c2f84de480/volumes" Nov 01 06:34:59.355732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:59.355689 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 01 06:34:59.356166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:34:59.355756 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 01 06:35:00.708067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:00.707779 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:35:00.708573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:00.708395 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:35:00.709225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:00.709204 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 06:35:02.113447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.113413 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w"] Nov 01 06:35:02.114050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.114034 4914 cpu_manager.go:414] "containerMap: removing container" podUID="38313716-cb5a-4fce-afc3-56c2f84de480" containerName="maas-api" Nov 01 06:35:02.114107 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.114053 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38313716-cb5a-4fce-afc3-56c2f84de480" containerName="maas-api" Nov 01 06:35:02.114159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.114150 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="38313716-cb5a-4fce-afc3-56c2f84de480" containerName="maas-api" Nov 01 06:35:02.118994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.118973 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.176866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.176836 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w"] Nov 01 06:35:02.193453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193420 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.193627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193475 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.193627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193610 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f55r\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.193742 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193643 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.193742 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193661 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.193855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193834 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.193912 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193894 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.194011 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.193994 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.194093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.194030 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.223390 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.223322 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-df0a0131-gw-openshift-default-dockercfg-4ptks\"" Nov 01 06:35:02.307596 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.307552 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.308393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.308367 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.308926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.308904 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.310227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.310208 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.311148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.311128 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.311750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.311733 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.314645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.312393 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.315119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.315102 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.315929 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.315892 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f55r\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.316234 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.310982 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.316393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.315023 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.316498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.308215 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.316585 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.311658 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.316673 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.308800 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.322095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.321215 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.323148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.322980 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.372617 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.372541 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.373982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.373960 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f55r\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r\") pod \"e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.432432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.432392 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:02.874538 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.874511 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w"] Nov 01 06:35:02.875835 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:35:02.875811 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd108a036_a6a8_4a7b_ba20_aa85c4d77a52.slice/crio-db07483fcdaa2349f7e8ab34dd72edd80fc6801501917998f74c658bb01efde1 WatchSource:0}: Error finding container db07483fcdaa2349f7e8ab34dd72edd80fc6801501917998f74c658bb01efde1: Status 404 returned error can't find the container with id db07483fcdaa2349f7e8ab34dd72edd80fc6801501917998f74c658bb01efde1 Nov 01 06:35:02.878419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.878386 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:35:02.878485 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.878465 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:35:02.878523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:02.878495 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:35:03.104718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.104674 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" event={"ID":"d108a036-a6a8-4a7b-ba20-aa85c4d77a52","Type":"ContainerStarted","Data":"150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079"} Nov 01 06:35:03.104904 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.104735 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" event={"ID":"d108a036-a6a8-4a7b-ba20-aa85c4d77a52","Type":"ContainerStarted","Data":"db07483fcdaa2349f7e8ab34dd72edd80fc6801501917998f74c658bb01efde1"} Nov 01 06:35:03.223125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.223010 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podStartSLOduration=1.222995141 podStartE2EDuration="1.222995141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:03.220809482 +0000 UTC m=+1518.773598678" watchObservedRunningTime="2026-11-01 06:35:03.222995141 +0000 UTC m=+1518.775784333" Nov 01 06:35:03.291848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.291808 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl"] Nov 01 06:35:03.298536 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.297749 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.345913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.345877 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.346070 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.346026 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8z9\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.346175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.346158 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.346235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.346201 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.352518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352490 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-2b38f786e9be\"" Nov 01 06:35:03.352736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352719 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\"" Nov 01 06:35:03.352832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352788 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\"" Nov 01 06:35:03.352911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352855 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-44f4ea1effdd\"" Nov 01 06:35:03.352911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352899 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\"" Nov 01 06:35:03.352911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352909 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-c8aede0ff036\"" Nov 01 06:35:03.353125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352965 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\"" Nov 01 06:35:03.353125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.352995 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\"" Nov 01 06:35:03.355780 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.355758 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl"] Nov 01 06:35:03.445305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.441978 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:03.451305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.450907 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:35:03.451305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.450987 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:35:03.456309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.454391 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.456309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.454541 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8z9\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.456309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.454645 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.456309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.454685 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.456309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.455542 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.462170 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.462148 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\"" Nov 01 06:35:03.462551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.462534 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:35:03.472841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.468223 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.474091 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.473957 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.517453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.517261 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8z9\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:03.621980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:03.621949 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:04.055256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:04.055227 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl"] Nov 01 06:35:04.057102 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:35:04.057059 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f8efa0_55e0_441f_90d5_68c347df8759.slice/crio-579a16238f49b8c6f215fea011a2d2a2134d0f07de47959a02cf807ed41b9e91 WatchSource:0}: Error finding container 579a16238f49b8c6f215fea011a2d2a2134d0f07de47959a02cf807ed41b9e91: Status 404 returned error can't find the container with id 579a16238f49b8c6f215fea011a2d2a2134d0f07de47959a02cf807ed41b9e91 Nov 01 06:35:04.112546 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:04.112521 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" event={"ID":"62f8efa0-55e0-441f-90d5-68c347df8759","Type":"ContainerStarted","Data":"579a16238f49b8c6f215fea011a2d2a2134d0f07de47959a02cf807ed41b9e91"} Nov 01 06:35:04.438189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:04.438163 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:05.120667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:05.120634 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" event={"ID":"62f8efa0-55e0-441f-90d5-68c347df8759","Type":"ContainerStarted","Data":"f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d"} Nov 01 06:35:05.120881 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:05.120857 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:05.121075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:05.121038 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:05.121801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:05.121781 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:35:05.237469 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:05.237412 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" podStartSLOduration=2.23739915 podStartE2EDuration="2.23739915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:05.234242568 +0000 UTC m=+1520.787031763" watchObservedRunningTime="2026-11-01 06:35:05.23739915 +0000 UTC m=+1520.790188343" Nov 01 06:35:06.590910 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.590876 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l"] Nov 01 06:35:06.595241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.595224 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.648649 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.648602 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-294b20a2cf15\"" Nov 01 06:35:06.683163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.683123 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:35:06.698617 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.698589 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25719d\"" Nov 01 06:35:06.698617 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.698612 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"openshift-service-ca.crt\"" Nov 01 06:35:06.698852 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.698684 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"kube-root-ca.crt\"" Nov 01 06:35:06.698852 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.698687 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-e2e--bf6d8971-dockercfg-6qmzh\"" Nov 01 06:35:06.698852 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.698693 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"routing-overlay\"" Nov 01 06:35:06.701801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.701775 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l"] Nov 01 06:35:06.716369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.716342 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.716498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.716391 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.716498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.716426 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.716616 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.716594 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kg2z\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.817841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.817810 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kg2z\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.817990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.817970 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.818034 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.817999 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.818034 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.818021 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.818921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.818877 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.818921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.818905 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.820628 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.820598 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.874830 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.874796 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kg2z\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:06.910226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:06.910197 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:07.411340 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:07.411312 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l"] Nov 01 06:35:07.412000 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:35:07.411973 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e836cc_bf7d_4391_9834_da55537b7bfd.slice/crio-92324680dfdf440535e60796943cba62e6ed9310b6d373ccef323d3d11b5de19 WatchSource:0}: Error finding container 92324680dfdf440535e60796943cba62e6ed9310b6d373ccef323d3d11b5de19: Status 404 returned error can't find the container with id 92324680dfdf440535e60796943cba62e6ed9310b6d373ccef323d3d11b5de19 Nov 01 06:35:07.593581 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:35:07.593543 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:35:08.159903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:08.159868 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" event={"ID":"16e836cc-bf7d-4391-9834-da55537b7bfd","Type":"ContainerStarted","Data":"55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f"} Nov 01 06:35:08.160056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:08.159921 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" event={"ID":"16e836cc-bf7d-4391-9834-da55537b7bfd","Type":"ContainerStarted","Data":"92324680dfdf440535e60796943cba62e6ed9310b6d373ccef323d3d11b5de19"} Nov 01 06:35:08.160056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:08.160041 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:08.273930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:08.273876 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" podStartSLOduration=2.273862089 podStartE2EDuration="2.273862089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:08.27235623 +0000 UTC m=+1523.825145424" watchObservedRunningTime="2026-11-01 06:35:08.273862089 +0000 UTC m=+1523.826651301" Nov 01 06:35:09.365055 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.365024 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt"] Nov 01 06:35:09.382857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.382821 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.395999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.395805 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt"] Nov 01 06:35:09.436014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.435982 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-df0a0131\"" Nov 01 06:35:09.436014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.435983 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-df0a0131-dockercfg-bqbnx\"" Nov 01 06:35:09.440668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.440640 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2s9f\" (UniqueName: \"kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f\") pod \"payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.440798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.440760 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.541808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.541769 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2s9f\" (UniqueName: \"kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f\") pod \"payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.541983 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.541851 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.542607 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.542589 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.598503 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.598473 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2s9f\" (UniqueName: \"kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f\") pod \"payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:09.705931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:09.705848 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:10.075806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:10.075778 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt"] Nov 01 06:35:10.076197 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:35:10.076169 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c3adc3_8ad2_4030_a920_bf3eba62c8e6.slice/crio-bedf55dd59b7afd79919ebfd5508a953dfba252e85deed02b8cd62941dffbf5f WatchSource:0}: Error finding container bedf55dd59b7afd79919ebfd5508a953dfba252e85deed02b8cd62941dffbf5f: Status 404 returned error can't find the container with id bedf55dd59b7afd79919ebfd5508a953dfba252e85deed02b8cd62941dffbf5f Nov 01 06:35:10.173837 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:10.173810 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" event={"ID":"28c3adc3-8ad2-4030-a920-bf3eba62c8e6","Type":"ContainerStarted","Data":"0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d"} Nov 01 06:35:10.173950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:10.173865 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" event={"ID":"28c3adc3-8ad2-4030-a920-bf3eba62c8e6","Type":"ContainerStarted","Data":"bedf55dd59b7afd79919ebfd5508a953dfba252e85deed02b8cd62941dffbf5f"} Nov 01 06:35:10.174064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:10.174044 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:10.286671 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:10.286621 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" podStartSLOduration=1.286608898 podStartE2EDuration="1.286608898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:10.286434053 +0000 UTC m=+1525.839223248" watchObservedRunningTime="2026-11-01 06:35:10.286608898 +0000 UTC m=+1525.839398090" Nov 01 06:35:11.182676 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.182644 4914 generic.go:356] "Generic (PLEG): container finished" podID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerID="fdf926a005a1faf40aaca9e206b2bb2fbf03d5e354150f7f32dd5dc93bb40624" exitCode=0 Nov 01 06:35:11.183078 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.182721 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" event={"ID":"747bf3f0-a5f9-4f6c-bf68-9e55849714a9","Type":"ContainerDied","Data":"fdf926a005a1faf40aaca9e206b2bb2fbf03d5e354150f7f32dd5dc93bb40624"} Nov 01 06:35:11.428769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.428709 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:35:11.503367 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.503338 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44\" (UniqueName: \"kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44\") pod \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " Nov 01 06:35:11.503518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.503417 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume\") pod \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\" (UID: \"747bf3f0-a5f9-4f6c-bf68-9e55849714a9\") " Nov 01 06:35:11.503812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.503789 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume" pod "747bf3f0-a5f9-4f6c-bf68-9e55849714a9" (UID: "747bf3f0-a5f9-4f6c-bf68-9e55849714a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:11.504039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.504024 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:11.505659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.505634 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44" pod "747bf3f0-a5f9-4f6c-bf68-9e55849714a9" (UID: "747bf3f0-a5f9-4f6c-bf68-9e55849714a9"). InnerVolumeSpecName "kube-api-access-g6h44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:11.605698 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:11.605655 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h44\" (UniqueName: \"kubernetes.io/projected/747bf3f0-a5f9-4f6c-bf68-9e55849714a9-kube-api-access-g6h44\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:12.211311 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.201794 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" event={"ID":"747bf3f0-a5f9-4f6c-bf68-9e55849714a9","Type":"ContainerDied","Data":"66196f80533d2f504ce20b739d8a7a0140a60538a3ceded415a46a4fcbeb230d"} Nov 01 06:35:12.211311 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.201870 4914 scope.go:122] "RemoveContainer" containerID="fdf926a005a1faf40aaca9e206b2bb2fbf03d5e354150f7f32dd5dc93bb40624" Nov 01 06:35:12.211311 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.202041 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk" Nov 01 06:35:12.352855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.352814 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b"] Nov 01 06:35:12.353518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.353498 4914 cpu_manager.go:414] "containerMap: removing container" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerName="payload-processing" Nov 01 06:35:12.353629 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.353519 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerName="payload-processing" Nov 01 06:35:12.353710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.353696 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" containerName="payload-processing" Nov 01 06:35:12.358473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.358452 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.380656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.380630 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk"] Nov 01 06:35:12.386207 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.386183 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7b2a3-ffb99b799-lcpkk"] Nov 01 06:35:12.411576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.411549 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ff1dc\"" Nov 01 06:35:12.411719 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.411614 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"openshift-service-ca.crt\"" Nov 01 06:35:12.413397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.413375 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b"] Nov 01 06:35:12.427828 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.427797 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"routing-overlay\"" Nov 01 06:35:12.427945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.427832 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-e2e--528f7081-dockercfg-c6b6l\"" Nov 01 06:35:12.427945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.427888 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-529628bc0fc6\"" Nov 01 06:35:12.428025 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.427957 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"kube-root-ca.crt\"" Nov 01 06:35:12.525163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.525075 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gb4\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.525163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.525137 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.525417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.525255 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.525417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.525375 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.617729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.617696 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747bf3f0-a5f9-4f6c-bf68-9e55849714a9" path="/var/lib/kubelet/pods/747bf3f0-a5f9-4f6c-bf68-9e55849714a9/volumes" Nov 01 06:35:12.626720 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.626686 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.626852 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.626819 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gb4\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.626932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.626897 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.627000 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.626982 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.631825 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.628052 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.631825 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.628138 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.637857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.637828 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.684519 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.684487 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gb4\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:12.972464 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:12.972430 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:13.345872 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:35:13.345842 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1792c887_eb33_4b49_8c73_8fd056b3d3e5.slice/crio-6d92dc0fcd40b5a67eaf9ea6f6b2973b4772ba5f3e5a9e02f514085403ceb405 WatchSource:0}: Error finding container 6d92dc0fcd40b5a67eaf9ea6f6b2973b4772ba5f3e5a9e02f514085403ceb405: Status 404 returned error can't find the container with id 6d92dc0fcd40b5a67eaf9ea6f6b2973b4772ba5f3e5a9e02f514085403ceb405 Nov 01 06:35:13.345872 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:13.345851 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b"] Nov 01 06:35:14.008342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.008268 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7"] Nov 01 06:35:14.008527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.008505 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" podUID="b221fcc9-bb3c-4694-ba54-57cb8c9e7047" containerName="maas-api" containerID="cri-o://f5a8036ddf471c74be9f4aab1fef62a4b9556d7c550be0588fc2263380796b36" gracePeriod=30 Nov 01 06:35:14.301966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.301882 4914 generic.go:356] "Generic (PLEG): container finished" podID="b221fcc9-bb3c-4694-ba54-57cb8c9e7047" containerID="f5a8036ddf471c74be9f4aab1fef62a4b9556d7c550be0588fc2263380796b36" exitCode=0 Nov 01 06:35:14.302118 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.302017 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" event={"ID":"b221fcc9-bb3c-4694-ba54-57cb8c9e7047","Type":"ContainerDied","Data":"f5a8036ddf471c74be9f4aab1fef62a4b9556d7c550be0588fc2263380796b36"} Nov 01 06:35:14.304114 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.304086 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" event={"ID":"1792c887-eb33-4b49-8c73-8fd056b3d3e5","Type":"ContainerStarted","Data":"20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef"} Nov 01 06:35:14.304338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.304320 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" event={"ID":"1792c887-eb33-4b49-8c73-8fd056b3d3e5","Type":"ContainerStarted","Data":"6d92dc0fcd40b5a67eaf9ea6f6b2973b4772ba5f3e5a9e02f514085403ceb405"} Nov 01 06:35:14.304477 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.304466 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:14.333456 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.333434 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:35:14.491992 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.491959 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs\") pod \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " Nov 01 06:35:14.492459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.492239 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca\") pod \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " Nov 01 06:35:14.492459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.492336 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7\" (UniqueName: \"kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7\") pod \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " Nov 01 06:35:14.492563 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.492512 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\") pod \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\" (UID: \"b221fcc9-bb3c-4694-ba54-57cb8c9e7047\") " Nov 01 06:35:14.492845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.492769 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca" pod "b221fcc9-bb3c-4694-ba54-57cb8c9e7047" (UID: "b221fcc9-bb3c-4694-ba54-57cb8c9e7047"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:35:14.493213 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.493187 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:14.494786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.494753 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs" pod "b221fcc9-bb3c-4694-ba54-57cb8c9e7047" (UID: "b221fcc9-bb3c-4694-ba54-57cb8c9e7047"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:35:14.495230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.495206 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7" pod "b221fcc9-bb3c-4694-ba54-57cb8c9e7047" (UID: "b221fcc9-bb3c-4694-ba54-57cb8c9e7047"). InnerVolumeSpecName "kube-api-access-tdtr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:35:14.495720 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.495698 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls" pod "b221fcc9-bb3c-4694-ba54-57cb8c9e7047" (UID: "b221fcc9-bb3c-4694-ba54-57cb8c9e7047"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:35:14.564399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.562565 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" podStartSLOduration=2.562540248 podStartE2EDuration="2.562540248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:35:14.435044353 +0000 UTC m=+1529.987833547" watchObservedRunningTime="2026-11-01 06:35:14.562540248 +0000 UTC m=+1530.115329442" Nov 01 06:35:14.594930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.594856 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:14.594930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.594887 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdtr7\" (UniqueName: \"kubernetes.io/projected/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-kube-api-access-tdtr7\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:14.594930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:14.594898 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b221fcc9-bb3c-4694-ba54-57cb8c9e7047-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:35:15.310240 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:15.310206 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" Nov 01 06:35:15.310465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:15.310243 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7" event={"ID":"b221fcc9-bb3c-4694-ba54-57cb8c9e7047","Type":"ContainerDied","Data":"a93df5bcf7adfd47f8b81f0b1e9e3828bf4495f1452ff56301f1de995e236a56"} Nov 01 06:35:15.310465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:15.310324 4914 scope.go:122] "RemoveContainer" containerID="f5a8036ddf471c74be9f4aab1fef62a4b9556d7c550be0588fc2263380796b36" Nov 01 06:35:15.475334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:15.475301 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7"] Nov 01 06:35:15.525726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:15.525691 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e7b2a3-85766bb65-wsnq7"] Nov 01 06:35:16.144648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:16.144618 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:35:16.617989 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:16.617949 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b221fcc9-bb3c-4694-ba54-57cb8c9e7047" path="/var/lib/kubelet/pods/b221fcc9-bb3c-4694-ba54-57cb8c9e7047/volumes" Nov 01 06:35:19.169258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:19.169228 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:35:21.186591 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:21.186555 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:35:25.313468 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:35:25.313435 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:35:37.599878 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:35:37.599840 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:36:07.593560 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:07.593478 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:36:12.285152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.285119 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l"] Nov 01 06:36:12.285572 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.285369 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" podUID="c4f3348f-1031-4609-a8ce-603f99af9b39" containerName="maas-api" containerID="cri-o://628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523" gracePeriod=30 Nov 01 06:36:12.638612 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.638585 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:36:12.734785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.734752 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh\" (UniqueName: \"kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh\") pod \"c4f3348f-1031-4609-a8ce-603f99af9b39\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " Nov 01 06:36:12.734966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.734859 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca\") pod \"c4f3348f-1031-4609-a8ce-603f99af9b39\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " Nov 01 06:36:12.734966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.734922 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs\") pod \"c4f3348f-1031-4609-a8ce-603f99af9b39\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " Nov 01 06:36:12.735086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.735049 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\") pod \"c4f3348f-1031-4609-a8ce-603f99af9b39\" (UID: \"c4f3348f-1031-4609-a8ce-603f99af9b39\") " Nov 01 06:36:12.735301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.735258 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca" pod "c4f3348f-1031-4609-a8ce-603f99af9b39" (UID: "c4f3348f-1031-4609-a8ce-603f99af9b39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:12.735923 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.735898 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f3348f-1031-4609-a8ce-603f99af9b39-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:12.737234 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.737210 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls" pod "c4f3348f-1031-4609-a8ce-603f99af9b39" (UID: "c4f3348f-1031-4609-a8ce-603f99af9b39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:36:12.737345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.737324 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs" pod "c4f3348f-1031-4609-a8ce-603f99af9b39" (UID: "c4f3348f-1031-4609-a8ce-603f99af9b39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:36:12.737417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.737404 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh" pod "c4f3348f-1031-4609-a8ce-603f99af9b39" (UID: "c4f3348f-1031-4609-a8ce-603f99af9b39"). InnerVolumeSpecName "kube-api-access-qszsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:12.742341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.742317 4914 generic.go:356] "Generic (PLEG): container finished" podID="c4f3348f-1031-4609-a8ce-603f99af9b39" containerID="628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523" exitCode=0 Nov 01 06:36:12.742428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.742365 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" Nov 01 06:36:12.742473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.742409 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" event={"ID":"c4f3348f-1031-4609-a8ce-603f99af9b39","Type":"ContainerDied","Data":"628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523"} Nov 01 06:36:12.742508 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.742487 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l" event={"ID":"c4f3348f-1031-4609-a8ce-603f99af9b39","Type":"ContainerDied","Data":"8ffa302b30336193e3d76f1f347545739484a0f1e3a2731ad568187c4ce87659"} Nov 01 06:36:12.742549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.742522 4914 scope.go:122] "RemoveContainer" containerID="628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523" Nov 01 06:36:12.767503 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.767478 4914 scope.go:122] "RemoveContainer" containerID="628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523" Nov 01 06:36:12.768000 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:12.767972 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523\": container with ID starting with 628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523 not found: ID does not exist" containerID="628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523" Nov 01 06:36:12.768102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.768008 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523"} err="failed to get container status \"628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523\": rpc error: code = NotFound desc = could not find container \"628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523\": container with ID starting with 628e774a3c2183523d099c28e67e09c8a39d357959303c041361fd7bf0c4f523 not found: ID does not exist" Nov 01 06:36:12.836843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.836813 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszsh\" (UniqueName: \"kubernetes.io/projected/c4f3348f-1031-4609-a8ce-603f99af9b39-kube-api-access-qszsh\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:12.836843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.836837 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:12.836843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.836849 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f3348f-1031-4609-a8ce-603f99af9b39-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:12.948342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.945569 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l"] Nov 01 06:36:12.948342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:12.945637 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-390a75-5d979d4f6f-jtf7l"] Nov 01 06:36:14.617313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:14.617268 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f3348f-1031-4609-a8ce-603f99af9b39" path="/var/lib/kubelet/pods/c4f3348f-1031-4609-a8ce-603f99af9b39/volumes" Nov 01 06:36:23.344447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:23.344407 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz"] Nov 01 06:36:23.344984 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:23.344604 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" containerID="cri-o://9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56" gracePeriod=30 Nov 01 06:36:25.459013 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:25.458974 4914 patch_prober.go:29] interesting pod/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:36:25.459418 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:25.459043 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:36:28.617575 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.617501 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:36:28.702404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702369 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702434 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702468 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702546 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702599 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702653 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:28.702755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702669 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702727 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702801 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.702932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702896 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs\") pod \"15ff29ad-3aaa-4333-b412-6a2bce942884\" (UID: \"15ff29ad-3aaa-4333-b412-6a2bce942884\") " Nov 01 06:36:28.703162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702894 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:28.703162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.702980 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:28.703545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.703185 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:28.703545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.703144 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:28.704263 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.704240 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.704876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.704803 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.704876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.704831 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.704876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.704845 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.704876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.704857 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ff29ad-3aaa-4333-b412-6a2bce942884-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.705145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.704992 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:28.705228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.705212 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:36:28.705346 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.705329 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "kube-api-access-s659r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:28.705491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.705464 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token" pod "15ff29ad-3aaa-4333-b412-6a2bce942884" (UID: "15ff29ad-3aaa-4333-b412-6a2bce942884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:28.806748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.806628 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s659r\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-kube-api-access-s659r\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.806748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.806669 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.806748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.806683 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.806748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.806695 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ff29ad-3aaa-4333-b412-6a2bce942884-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:28.944697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.937685 4914 generic.go:356] "Generic (PLEG): container finished" podID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerID="9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56" exitCode=0 Nov 01 06:36:28.944697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.937831 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" event={"ID":"15ff29ad-3aaa-4333-b412-6a2bce942884","Type":"ContainerDied","Data":"9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56"} Nov 01 06:36:28.944697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.937879 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" event={"ID":"15ff29ad-3aaa-4333-b412-6a2bce942884","Type":"ContainerDied","Data":"86fe71f280b168010b32f4910f3ba5a0d71333b3bbf9e4a5e253771d71e29115"} Nov 01 06:36:28.944697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.937907 4914 scope.go:122] "RemoveContainer" containerID="9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56" Nov 01 06:36:28.944697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.938067 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz" Nov 01 06:36:28.964385 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.964145 4914 scope.go:122] "RemoveContainer" containerID="e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8" Nov 01 06:36:28.976064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.976042 4914 scope.go:122] "RemoveContainer" containerID="9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56" Nov 01 06:36:28.976476 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:28.976448 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56\": container with ID starting with 9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56 not found: ID does not exist" containerID="9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56" Nov 01 06:36:28.976573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.976481 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56"} err="failed to get container status \"9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56\": rpc error: code = NotFound desc = could not find container \"9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56\": container with ID starting with 9c2df75f71e87d4c833acc024d93f6571e479586b8902b39bbbf09786dd6fd56 not found: ID does not exist" Nov 01 06:36:28.976573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.976507 4914 scope.go:122] "RemoveContainer" containerID="e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8" Nov 01 06:36:28.976770 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:28.976754 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8\": container with ID starting with e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8 not found: ID does not exist" containerID="e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8" Nov 01 06:36:28.976822 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:28.976782 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8"} err="failed to get container status \"e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8\": rpc error: code = NotFound desc = could not find container \"e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8\": container with ID starting with e24cf61c9d6fd4453434f9051403a8ecccde4f8cbc35ea4ffba0a7fae7fb33a8 not found: ID does not exist" Nov 01 06:36:29.107346 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:29.107309 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz"] Nov 01 06:36:29.112709 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:29.112681 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-390a75-openshift-default-856d5799f7-j9ztz"] Nov 01 06:36:30.587776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.587739 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2"] Nov 01 06:36:30.589266 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.589222 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" containerID="cri-o://15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037" gracePeriod=30 Nov 01 06:36:30.617288 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.617242 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" path="/var/lib/kubelet/pods/15ff29ad-3aaa-4333-b412-6a2bce942884/volumes" Nov 01 06:36:30.637090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.637059 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49"] Nov 01 06:36:30.637362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.637338 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" containerID="cri-o://55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c" gracePeriod=30 Nov 01 06:36:30.639533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.639513 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr"] Nov 01 06:36:30.639691 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:30.639673 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" containerID="cri-o://8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59" gracePeriod=30 Nov 01 06:36:32.070667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.070634 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v"] Nov 01 06:36:32.071227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071214 4914 cpu_manager.go:414] "containerMap: removing container" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" Nov 01 06:36:32.071307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071229 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" Nov 01 06:36:32.071307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071240 4914 cpu_manager.go:414] "containerMap: removing container" podUID="b221fcc9-bb3c-4694-ba54-57cb8c9e7047" containerName="maas-api" Nov 01 06:36:32.071307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071246 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b221fcc9-bb3c-4694-ba54-57cb8c9e7047" containerName="maas-api" Nov 01 06:36:32.071307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071301 4914 cpu_manager.go:414] "containerMap: removing container" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" Nov 01 06:36:32.071307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071307 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" Nov 01 06:36:32.071467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071322 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c4f3348f-1031-4609-a8ce-603f99af9b39" containerName="maas-api" Nov 01 06:36:32.071467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071326 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f3348f-1031-4609-a8ce-603f99af9b39" containerName="maas-api" Nov 01 06:36:32.071467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071414 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" Nov 01 06:36:32.071467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071423 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f3348f-1031-4609-a8ce-603f99af9b39" containerName="maas-api" Nov 01 06:36:32.071467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071430 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ff29ad-3aaa-4333-b412-6a2bce942884" containerName="istio-proxy" Nov 01 06:36:32.071467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.071436 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b221fcc9-bb3c-4694-ba54-57cb8c9e7047" containerName="maas-api" Nov 01 06:36:32.076047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.076029 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.131240 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.131208 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v"] Nov 01 06:36:32.238776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.226039 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkjw\" (UniqueName: \"kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw\") pod \"payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.283624 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.283582 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.312300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.234428 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2e0116-dockercfg-77q4f\"" Nov 01 06:36:32.312300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.234311 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2e0116\"" Nov 01 06:36:32.324367 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.320260 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6"] Nov 01 06:36:32.325628 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.325600 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.332108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.332081 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6"] Nov 01 06:36:32.410080 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.410044 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.410261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.410104 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkjw\" (UniqueName: \"kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw\") pod \"payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.410261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.410234 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24q6x\" (UniqueName: \"kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x\") pod \"payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.410440 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.410412 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.411014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.410995 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.466914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.466875 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkjw\" (UniqueName: \"kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw\") pod \"payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:32.511180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.511145 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.511347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.511248 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24q6x\" (UniqueName: \"kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x\") pod \"payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.511835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.511814 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.568534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.568504 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24q6x\" (UniqueName: \"kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x\") pod \"payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.650982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.650947 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:32.689587 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:32.689550 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:33.026899 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.026868 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6"] Nov 01 06:36:33.027819 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:36:33.027795 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c472150_761c_43e0_b003_b323c493d208.slice/crio-08fa4d1c0c16143e743fc682e6f674553faac868b7a5e92be72d120075d61e6d WatchSource:0}: Error finding container 08fa4d1c0c16143e743fc682e6f674553faac868b7a5e92be72d120075d61e6d: Status 404 returned error can't find the container with id 08fa4d1c0c16143e743fc682e6f674553faac868b7a5e92be72d120075d61e6d Nov 01 06:36:33.125847 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.125819 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v"] Nov 01 06:36:33.127006 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:36:33.126981 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b37d6d6_e5b3_465d_82c0_9a340cacd010.slice/crio-96cee406ad09632d90858881cde3dba8da3388f53e355700f1fce27d5887f55c WatchSource:0}: Error finding container 96cee406ad09632d90858881cde3dba8da3388f53e355700f1fce27d5887f55c: Status 404 returned error can't find the container with id 96cee406ad09632d90858881cde3dba8da3388f53e355700f1fce27d5887f55c Nov 01 06:36:33.971201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.971164 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" event={"ID":"0c472150-761c-43e0-b003-b323c493d208","Type":"ContainerStarted","Data":"6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3"} Nov 01 06:36:33.971430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.971233 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:33.971430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.971259 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" event={"ID":"0c472150-761c-43e0-b003-b323c493d208","Type":"ContainerStarted","Data":"08fa4d1c0c16143e743fc682e6f674553faac868b7a5e92be72d120075d61e6d"} Nov 01 06:36:33.972832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.972794 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" event={"ID":"7b37d6d6-e5b3-465d-82c0-9a340cacd010","Type":"ContainerStarted","Data":"f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1"} Nov 01 06:36:33.972945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.972841 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" event={"ID":"7b37d6d6-e5b3-465d-82c0-9a340cacd010","Type":"ContainerStarted","Data":"96cee406ad09632d90858881cde3dba8da3388f53e355700f1fce27d5887f55c"} Nov 01 06:36:33.972945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:33.972870 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:34.085298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:34.085233 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" podStartSLOduration=2.085220062 podStartE2EDuration="2.085220062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:34.08382314 +0000 UTC m=+1609.636612333" watchObservedRunningTime="2026-11-01 06:36:34.085220062 +0000 UTC m=+1609.638009254" Nov 01 06:36:34.226361 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:34.201959 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" podStartSLOduration=2.201937197 podStartE2EDuration="2.201937197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:36:34.199699717 +0000 UTC m=+1609.752488909" watchObservedRunningTime="2026-11-01 06:36:34.201937197 +0000 UTC m=+1609.754726391" Nov 01 06:36:35.960173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.960148 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:36:35.964341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.964318 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:36:35.982749 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.982719 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:36:35.985776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.985751 4914 generic.go:356] "Generic (PLEG): container finished" podID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerID="55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c" exitCode=0 Nov 01 06:36:35.985897 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.985794 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" Nov 01 06:36:35.985897 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.985833 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" event={"ID":"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa","Type":"ContainerDied","Data":"55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c"} Nov 01 06:36:35.985990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.985890 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49" event={"ID":"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa","Type":"ContainerDied","Data":"d4ddff270a87d30d607ca41b69baa0a4d484b69088f31358cd5859c784955766"} Nov 01 06:36:35.985990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.985946 4914 scope.go:122] "RemoveContainer" containerID="55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c" Nov 01 06:36:35.987176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.987155 4914 generic.go:356] "Generic (PLEG): container finished" podID="932b4994-7d4c-4b6a-9832-c6a179828959" containerID="15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037" exitCode=0 Nov 01 06:36:35.987261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.987203 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" Nov 01 06:36:35.987261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.987236 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" event={"ID":"932b4994-7d4c-4b6a-9832-c6a179828959","Type":"ContainerDied","Data":"15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037"} Nov 01 06:36:35.987402 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.987312 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2" event={"ID":"932b4994-7d4c-4b6a-9832-c6a179828959","Type":"ContainerDied","Data":"ce4663bb997db99b3c878bb55d7211b21ce432ab605a625ddf2b3269072e8a54"} Nov 01 06:36:35.988708 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.988689 4914 generic.go:356] "Generic (PLEG): container finished" podID="65e78e54-59de-4cad-ad4e-dda46617a274" containerID="8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59" exitCode=0 Nov 01 06:36:35.988798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.988728 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" Nov 01 06:36:35.988798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.988744 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" event={"ID":"65e78e54-59de-4cad-ad4e-dda46617a274","Type":"ContainerDied","Data":"8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59"} Nov 01 06:36:35.988798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.988776 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr" event={"ID":"65e78e54-59de-4cad-ad4e-dda46617a274","Type":"ContainerDied","Data":"7c430a8a811e4b4bf50858d601c95cfbc9c133a5dc0a108550c6a57d3b8dfddf"} Nov 01 06:36:35.995953 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.995929 4914 scope.go:122] "RemoveContainer" containerID="55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c" Nov 01 06:36:35.996196 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:35.996178 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c\": container with ID starting with 55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c not found: ID does not exist" containerID="55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c" Nov 01 06:36:35.996248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.996204 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c"} err="failed to get container status \"55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c\": rpc error: code = NotFound desc = could not find container \"55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c\": container with ID starting with 55567fb155442583110d16acd0e0442b64fb6c4dbd8518ad46c24e7f79351d9c not found: ID does not exist" Nov 01 06:36:35.996248 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:35.996219 4914 scope.go:122] "RemoveContainer" containerID="15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037" Nov 01 06:36:36.004790 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.004766 4914 scope.go:122] "RemoveContainer" containerID="15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037" Nov 01 06:36:36.005039 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:36.005020 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037\": container with ID starting with 15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037 not found: ID does not exist" containerID="15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037" Nov 01 06:36:36.005090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.005045 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037"} err="failed to get container status \"15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037\": rpc error: code = NotFound desc = could not find container \"15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037\": container with ID starting with 15d2cff29067282d691b093e4fe06781ac40a13cb1ce004bd0b2956cb613b037 not found: ID does not exist" Nov 01 06:36:36.005090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.005060 4914 scope.go:122] "RemoveContainer" containerID="8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59" Nov 01 06:36:36.015804 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.015783 4914 scope.go:122] "RemoveContainer" containerID="8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59" Nov 01 06:36:36.016194 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:36.016174 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59\": container with ID starting with 8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59 not found: ID does not exist" containerID="8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59" Nov 01 06:36:36.016264 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.016204 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59"} err="failed to get container status \"8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59\": rpc error: code = NotFound desc = could not find container \"8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59\": container with ID starting with 8fee56d5f5cc7cc4291bcd61ee69b61de67c49634fab747a12f025e9d5b7fe59 not found: ID does not exist" Nov 01 06:36:36.080915 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.080890 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.081096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.080941 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.081096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.080971 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.081214 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081139 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.081214 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081204 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.081349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081225 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.081349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081304 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.081454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081391 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.081564 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081545 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.081680 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081608 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.081680 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081667 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:36.081789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081647 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:36.081856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081667 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.081993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081976 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.082140 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082114 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.082257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.081851 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.082257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082153 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.082257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082173 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082253 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082332 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082370 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082371 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082409 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082453 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082511 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082536 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082581 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.082664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082613 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.083176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082748 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.083176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082912 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data\") pod \"932b4994-7d4c-4b6a-9832-c6a179828959\" (UID: \"932b4994-7d4c-4b6a-9832-c6a179828959\") " Nov 01 06:36:36.083176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082957 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.083176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.082989 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.083176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083017 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo\") pod \"65e78e54-59de-4cad-ad4e-dda46617a274\" (UID: \"65e78e54-59de-4cad-ad4e-dda46617a274\") " Nov 01 06:36:36.083176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083060 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs\") pod \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\" (UID: \"f2f842fa-4ca0-4804-af3e-c84ba34bf6aa\") " Nov 01 06:36:36.083743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083666 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.083743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083690 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65e78e54-59de-4cad-ad4e-dda46617a274-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.083743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083703 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932b4994-7d4c-4b6a-9832-c6a179828959-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.083743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083715 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.083743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083728 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.083743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083740 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.084062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083754 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.084062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.083942 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "kube-api-access-n96vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:36.084062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084006 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.084062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084008 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "kube-api-access-4m6tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:36.084062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084034 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.084338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084114 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.084338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084256 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.084338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084324 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.084578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084555 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.085383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084751 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "kube-api-access-bfxqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:36.085383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.084826 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.085383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.085181 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:36.085383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.085260 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:36.085383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.085305 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.085383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.085353 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:36:36.086599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.086534 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token" pod "932b4994-7d4c-4b6a-9832-c6a179828959" (UID: "932b4994-7d4c-4b6a-9832-c6a179828959"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:36.087214 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.087188 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:36:36.087447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.087373 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:36:36.087447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.087376 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy" pod "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" (UID: "f2f842fa-4ca0-4804-af3e-c84ba34bf6aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.087447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.087389 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:36:36.087569 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.087460 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token" pod "65e78e54-59de-4cad-ad4e-dda46617a274" (UID: "65e78e54-59de-4cad-ad4e-dda46617a274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184425 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184452 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184461 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184472 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184480 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65e78e54-59de-4cad-ad4e-dda46617a274-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184488 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184497 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96vc\" (UniqueName: \"kubernetes.io/projected/65e78e54-59de-4cad-ad4e-dda46617a274-kube-api-access-n96vc\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184505 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184507 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m6tq\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-kube-api-access-4m6tq\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184515 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184524 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxqp\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-kube-api-access-bfxqp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184532 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932b4994-7d4c-4b6a-9832-c6a179828959-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184542 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184551 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184558 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184566 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184574 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184582 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184590 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932b4994-7d4c-4b6a-9832-c6a179828959-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184598 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65e78e54-59de-4cad-ad4e-dda46617a274-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.184848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.184606 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932b4994-7d4c-4b6a-9832-c6a179828959-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:36.531329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.531065 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr"] Nov 01 06:36:36.535798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.535771 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-2ggdr"] Nov 01 06:36:36.618070 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.618035 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" path="/var/lib/kubelet/pods/65e78e54-59de-4cad-ad4e-dda46617a274/volumes" Nov 01 06:36:36.646119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.646089 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49"] Nov 01 06:36:36.650127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.650101 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-5sl49"] Nov 01 06:36:36.759231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.759195 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2"] Nov 01 06:36:36.762785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:36.762755 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-qqlv2"] Nov 01 06:36:37.593977 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:36:37.593938 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:36:38.617622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:38.617588 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" path="/var/lib/kubelet/pods/932b4994-7d4c-4b6a-9832-c6a179828959/volumes" Nov 01 06:36:38.618069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:38.618055 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" path="/var/lib/kubelet/pods/f2f842fa-4ca0-4804-af3e-c84ba34bf6aa/volumes" Nov 01 06:36:44.981922 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:44.981890 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 06:36:44.982331 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:44.982075 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 06:36:54.231079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:54.231046 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt"] Nov 01 06:36:59.231611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.231569 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" podUID="28c3adc3-8ad2-4030-a920-bf3eba62c8e6" containerName="payload-processing" containerID="cri-o://0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d" gracePeriod=25 Nov 01 06:36:59.536997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.536977 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:36:59.684842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.684814 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f\" (UniqueName: \"kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f\") pod \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " Nov 01 06:36:59.685017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.684993 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume\") pod \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\" (UID: \"28c3adc3-8ad2-4030-a920-bf3eba62c8e6\") " Nov 01 06:36:59.685378 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.685351 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume" pod "28c3adc3-8ad2-4030-a920-bf3eba62c8e6" (UID: "28c3adc3-8ad2-4030-a920-bf3eba62c8e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:36:59.687038 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.687015 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f" pod "28c3adc3-8ad2-4030-a920-bf3eba62c8e6" (UID: "28c3adc3-8ad2-4030-a920-bf3eba62c8e6"). InnerVolumeSpecName "kube-api-access-l2s9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:36:59.787023 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.786945 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:36:59.787023 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:36:59.786977 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2s9f\" (UniqueName: \"kubernetes.io/projected/28c3adc3-8ad2-4030-a920-bf3eba62c8e6-kube-api-access-l2s9f\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:00.127467 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.127432 4914 generic.go:356] "Generic (PLEG): container finished" podID="28c3adc3-8ad2-4030-a920-bf3eba62c8e6" containerID="0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d" exitCode=0 Nov 01 06:37:00.127668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.127485 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" Nov 01 06:37:00.127668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.127505 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" event={"ID":"28c3adc3-8ad2-4030-a920-bf3eba62c8e6","Type":"ContainerDied","Data":"0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d"} Nov 01 06:37:00.127668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.127561 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt" event={"ID":"28c3adc3-8ad2-4030-a920-bf3eba62c8e6","Type":"ContainerDied","Data":"bedf55dd59b7afd79919ebfd5508a953dfba252e85deed02b8cd62941dffbf5f"} Nov 01 06:37:00.127668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.127582 4914 scope.go:122] "RemoveContainer" containerID="0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d" Nov 01 06:37:00.137211 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.137190 4914 scope.go:122] "RemoveContainer" containerID="0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d" Nov 01 06:37:00.137527 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:00.137509 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d\": container with ID starting with 0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d not found: ID does not exist" containerID="0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d" Nov 01 06:37:00.137600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.137538 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d"} err="failed to get container status \"0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d\": rpc error: code = NotFound desc = could not find container \"0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d\": container with ID starting with 0dcce3c80b805f18ab874c6a2e361306944237adb6b2af9fde9533686551972d not found: ID does not exist" Nov 01 06:37:00.291045 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.291009 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt"] Nov 01 06:37:00.341039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.341005 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-df0a0131-656d84bdb9-4kjjt"] Nov 01 06:37:00.616821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:00.616782 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c3adc3-8ad2-4030-a920-bf3eba62c8e6" path="/var/lib/kubelet/pods/28c3adc3-8ad2-4030-a920-bf3eba62c8e6/volumes" Nov 01 06:37:05.122130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.122101 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:37:05.122548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.122150 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:37:05.158225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.158197 4914 generic.go:356] "Generic (PLEG): container finished" podID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerID="150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079" exitCode=0 Nov 01 06:37:05.158354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.158268 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" event={"ID":"d108a036-a6a8-4a7b-ba20-aa85c4d77a52","Type":"ContainerDied","Data":"150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079"} Nov 01 06:37:05.158615 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.158602 4914 scope.go:122] "RemoveContainer" containerID="150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079" Nov 01 06:37:05.159875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.159847 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:37:05.159932 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.159913 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:37:05.159968 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:05.159945 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:37:06.176913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:06.176864 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" event={"ID":"d108a036-a6a8-4a7b-ba20-aa85c4d77a52","Type":"ContainerStarted","Data":"431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9"} Nov 01 06:37:06.433373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:06.433268 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:37:06.433523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:06.433440 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:37:06.433523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:06.433486 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:37:07.433526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:07.433492 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:37:07.433891 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:07.433546 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:37:07.597487 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:07.597454 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:37:08.433136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:08.433107 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:37:08.433372 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:08.433156 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:37:09.433798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:09.433760 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:37:09.434168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:09.433829 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:37:10.433900 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:10.433869 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 01 06:37:10.434394 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:10.433924 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 01 06:37:11.346078 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346041 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg"] Nov 01 06:37:11.346705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346684 4914 cpu_manager.go:414] "containerMap: removing container" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" Nov 01 06:37:11.346705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346705 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" Nov 01 06:37:11.346875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346771 4914 cpu_manager.go:414] "containerMap: removing container" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" Nov 01 06:37:11.346875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346781 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" Nov 01 06:37:11.346875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346810 4914 cpu_manager.go:414] "containerMap: removing container" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" Nov 01 06:37:11.346875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346818 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" Nov 01 06:37:11.346875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346834 4914 cpu_manager.go:414] "containerMap: removing container" podUID="28c3adc3-8ad2-4030-a920-bf3eba62c8e6" containerName="payload-processing" Nov 01 06:37:11.346875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346842 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c3adc3-8ad2-4030-a920-bf3eba62c8e6" containerName="payload-processing" Nov 01 06:37:11.347148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.346994 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f842fa-4ca0-4804-af3e-c84ba34bf6aa" containerName="istio-proxy" Nov 01 06:37:11.347148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.347009 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e78e54-59de-4cad-ad4e-dda46617a274" containerName="istio-proxy" Nov 01 06:37:11.347148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.347022 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c3adc3-8ad2-4030-a920-bf3eba62c8e6" containerName="payload-processing" Nov 01 06:37:11.347148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.347035 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="932b4994-7d4c-4b6a-9832-c6a179828959" containerName="istio-proxy" Nov 01 06:37:11.350407 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.350386 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.403858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.403837 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2e0116-serving-cert\"" Nov 01 06:37:11.435203 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.435177 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.435553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.435307 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4n8\" (UniqueName: \"kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.435553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.435484 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.435639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.435597 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.437973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.437953 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:37:11.438259 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.438242 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:37:11.438903 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.438885 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:37:11.456324 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.456228 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg"] Nov 01 06:37:11.537155 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.537129 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.537301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.537211 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.537301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.537252 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.537415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.537319 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4n8\" (UniqueName: \"kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.537597 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:11.537578 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-serving-cert: secret "maas-api-e2e-shared-a-w3-2e0116-serving-cert" not found Nov 01 06:37:11.537665 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:11.537653 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls podName:5af8892c-4390-43c2-af8e-8a898aed5b54 nodeName:}" failed. No retries permitted until 2026-11-01 06:37:12.037635099 +0000 UTC m=+1647.590424290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" (UID: "5af8892c-4390-43c2-af8e-8a898aed5b54") : secret "maas-api-e2e-shared-a-w3-2e0116-serving-cert" not found Nov 01 06:37:11.537991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.537973 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.539645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.539629 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:11.663115 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:11.663035 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4n8\" (UniqueName: \"kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:12.057347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:12.057250 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:12.060185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:12.060163 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:12.263168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:12.263138 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:12.636237 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:37:12.636208 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af8892c_4390_43c2_af8e_8a898aed5b54.slice/crio-95442d25ab872506a08a879814185e44e1c56702873953e8bb874f64c2991b9c WatchSource:0}: Error finding container 95442d25ab872506a08a879814185e44e1c56702873953e8bb874f64c2991b9c: Status 404 returned error can't find the container with id 95442d25ab872506a08a879814185e44e1c56702873953e8bb874f64c2991b9c Nov 01 06:37:12.636237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:12.636213 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg"] Nov 01 06:37:13.228368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:13.228267 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" event={"ID":"5af8892c-4390-43c2-af8e-8a898aed5b54","Type":"ContainerStarted","Data":"95442d25ab872506a08a879814185e44e1c56702873953e8bb874f64c2991b9c"} Nov 01 06:37:14.235794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.235765 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" event={"ID":"5af8892c-4390-43c2-af8e-8a898aed5b54","Type":"ContainerStarted","Data":"1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c"} Nov 01 06:37:14.236194 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.235952 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:14.350145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.350106 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" podStartSLOduration=2.440690568 podStartE2EDuration="3.350094163s" totalImagesPullingTime="909.403595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:14.349389281 +0000 UTC m=+1649.902178475" watchObservedRunningTime="2026-11-01 06:37:14.350094163 +0000 UTC m=+1649.902883356" Nov 01 06:37:14.604260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.604227 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc"] Nov 01 06:37:14.608144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.608129 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.662632 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.662611 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-d41a0e39961b\"" Nov 01 06:37:14.674158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.674130 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc"] Nov 01 06:37:14.704534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.704481 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.704768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.704731 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.705071 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.705047 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6cf\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.705174 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.705132 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.806028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.806000 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.806121 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.806052 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.806162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.806144 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.806197 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.806187 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6cf\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.806586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.806565 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.806829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.806810 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.808700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.808683 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.862795 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.862743 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6cf\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:14.922005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:14.921983 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:15.290971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:15.290945 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc"] Nov 01 06:37:16.249669 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:16.249635 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" event={"ID":"1e095a19-cee8-4b1d-81e9-022d98de1cd9","Type":"ContainerStarted","Data":"0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635"} Nov 01 06:37:16.249838 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:16.249721 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" event={"ID":"1e095a19-cee8-4b1d-81e9-022d98de1cd9","Type":"ContainerStarted","Data":"0a2915c8d88a0dfe11bd9c9861d50ca9f1d36c83bf8a395379875b337d8c6175"} Nov 01 06:37:16.249838 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:16.249740 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:16.366516 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:16.366467 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" podStartSLOduration=2.366454271 podStartE2EDuration="2.366454271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:16.365176542 +0000 UTC m=+1651.917965738" watchObservedRunningTime="2026-11-01 06:37:16.366454271 +0000 UTC m=+1651.919243492" Nov 01 06:37:17.130333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.130297 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn"] Nov 01 06:37:17.134592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.134574 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.187860 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.187833 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-97c3302b7889\"" Nov 01 06:37:17.190983 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.190961 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn"] Nov 01 06:37:17.230536 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.230513 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.230665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.230558 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wwn\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.230665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.230582 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.230773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.230749 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.331381 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.331354 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.331490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.331400 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wwn\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.331490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.331423 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.331707 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.331672 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.332143 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.332119 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.332223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.332126 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.334347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.334330 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.387687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.387637 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wwn\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.449804 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.449783 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:17.824123 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:37:17.824096 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d69385c_7a3b_423f_85d1_36c257039dd8.slice/crio-a2520a43b24ef1b4dcba53a0e975826a36a2a06f0ac850e063d0a4ee5f6c0d21 WatchSource:0}: Error finding container a2520a43b24ef1b4dcba53a0e975826a36a2a06f0ac850e063d0a4ee5f6c0d21: Status 404 returned error can't find the container with id a2520a43b24ef1b4dcba53a0e975826a36a2a06f0ac850e063d0a4ee5f6c0d21 Nov 01 06:37:17.824123 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:17.824106 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn"] Nov 01 06:37:18.263340 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:18.263307 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" event={"ID":"8d69385c-7a3b-423f-85d1-36c257039dd8","Type":"ContainerStarted","Data":"f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c"} Nov 01 06:37:18.263501 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:18.263361 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" event={"ID":"8d69385c-7a3b-423f-85d1-36c257039dd8","Type":"ContainerStarted","Data":"a2520a43b24ef1b4dcba53a0e975826a36a2a06f0ac850e063d0a4ee5f6c0d21"} Nov 01 06:37:18.263501 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:18.263381 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:18.378027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:18.377983 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" podStartSLOduration=1.37797156 podStartE2EDuration="1.37797156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:18.376205188 +0000 UTC m=+1653.928994382" watchObservedRunningTime="2026-11-01 06:37:18.37797156 +0000 UTC m=+1653.930760752" Nov 01 06:37:20.246387 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:20.246357 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 06:37:22.761658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:22.761614 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w"] Nov 01 06:37:22.762066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:22.761921 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" containerID="cri-o://431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9" gracePeriod=30 Nov 01 06:37:23.922004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.921973 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz"] Nov 01 06:37:23.926218 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.926201 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.966778 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.966739 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.967160 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.967130 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.967412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.967395 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.967590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.967573 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.967730 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.967716 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.967952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.967912 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.968108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.968092 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr5m\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.968322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.968305 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:23.968502 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:23.968488 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.060008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.058023 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz"] Nov 01 06:37:24.060008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.058871 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8b4338ad-openshift-default-dockercfg-lnf5r\"" Nov 01 06:37:24.069520 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.069481 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.069752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.069730 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.069951 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.069932 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.070068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.070054 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr5m\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.070241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.070223 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.070621 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.070600 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.070747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.070731 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.070889 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.070872 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.071041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.071025 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.071601 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.071573 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.072238 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.072212 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.072457 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.072436 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.072635 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.072606 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.072793 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.072719 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.073733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.073706 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.075461 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.075429 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.169310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.168332 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr5m\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.175154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.175054 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token\") pod \"e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.241850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.241822 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:24.666797 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:37:24.666769 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8304a7_c449_48f1_9545_ac3e732eba47.slice/crio-f58098317a40ef70db4b63500880736317c7187825bc1d7640854529251287e9 WatchSource:0}: Error finding container f58098317a40ef70db4b63500880736317c7187825bc1d7640854529251287e9: Status 404 returned error can't find the container with id f58098317a40ef70db4b63500880736317c7187825bc1d7640854529251287e9 Nov 01 06:37:24.666911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.666797 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz"] Nov 01 06:37:24.669095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.669065 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:37:24.669195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.669133 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:37:24.669195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:24.669174 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:37:25.308499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:25.308466 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" event={"ID":"5e8304a7-c449-48f1-9545-ac3e732eba47","Type":"ContainerStarted","Data":"8bd3b58a90cbfc2b526a1867bc09dc3cc7c3722813aad81728a71959074cf8d1"} Nov 01 06:37:25.308940 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:25.308540 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" event={"ID":"5e8304a7-c449-48f1-9545-ac3e732eba47","Type":"ContainerStarted","Data":"f58098317a40ef70db4b63500880736317c7187825bc1d7640854529251287e9"} Nov 01 06:37:25.425566 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:25.425518 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podStartSLOduration=2.425505175 podStartE2EDuration="2.425505175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:25.423564357 +0000 UTC m=+1660.976353544" watchObservedRunningTime="2026-11-01 06:37:25.425505175 +0000 UTC m=+1660.978294368" Nov 01 06:37:26.242729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:26.242654 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:26.247667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:26.247646 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:26.314366 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:26.314343 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:26.315111 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:26.315091 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:37:26.440382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:26.440357 4914 patch_prober.go:29] interesting pod/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 01 06:37:26.440492 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:26.440402 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:37:27.258762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:27.258729 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:37:27.427431 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:27.427399 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l"] Nov 01 06:37:28.023492 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.023470 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:37:28.107408 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107379 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107434 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107457 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107487 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107551 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107620 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107686 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107726 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107728 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:28.107855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107766 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo\") pod \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\" (UID: \"d108a036-a6a8-4a7b-ba20-aa85c4d77a52\") " Nov 01 06:37:28.107855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107812 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:28.108054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.107994 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:28.108198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108143 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:28.108384 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108364 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:28.108625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108603 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.108690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108633 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.108690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108649 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.108690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108663 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.108690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.108678 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.109758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.109733 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:28.109854 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.109788 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:37:28.110233 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.110216 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:37:28.110312 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.110232 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r" pod "d108a036-a6a8-4a7b-ba20-aa85c4d77a52" (UID: "d108a036-a6a8-4a7b-ba20-aa85c4d77a52"). InnerVolumeSpecName "kube-api-access-5f55r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:28.209351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.209317 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.209351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.209341 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.209351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.209350 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.209579 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.209359 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f55r\" (UniqueName: \"kubernetes.io/projected/d108a036-a6a8-4a7b-ba20-aa85c4d77a52-kube-api-access-5f55r\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:28.328113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.328085 4914 generic.go:356] "Generic (PLEG): container finished" podID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerID="431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9" exitCode=0 Nov 01 06:37:28.328241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.328142 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" Nov 01 06:37:28.328241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.328172 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" event={"ID":"d108a036-a6a8-4a7b-ba20-aa85c4d77a52","Type":"ContainerDied","Data":"431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9"} Nov 01 06:37:28.328241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.328223 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w" event={"ID":"d108a036-a6a8-4a7b-ba20-aa85c4d77a52","Type":"ContainerDied","Data":"db07483fcdaa2349f7e8ab34dd72edd80fc6801501917998f74c658bb01efde1"} Nov 01 06:37:28.328425 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.328245 4914 scope.go:122] "RemoveContainer" containerID="431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9" Nov 01 06:37:28.339858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.339841 4914 scope.go:122] "RemoveContainer" containerID="150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079" Nov 01 06:37:28.351258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.349050 4914 scope.go:122] "RemoveContainer" containerID="431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9" Nov 01 06:37:28.351258 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:28.349474 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9\": container with ID starting with 431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9 not found: ID does not exist" containerID="431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9" Nov 01 06:37:28.351258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.349504 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9"} err="failed to get container status \"431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9\": rpc error: code = NotFound desc = could not find container \"431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9\": container with ID starting with 431b2b3b3d5f631348904ae1fd04c3353609c2670999740ec401665164fbc7d9 not found: ID does not exist" Nov 01 06:37:28.351258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.349528 4914 scope.go:122] "RemoveContainer" containerID="150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079" Nov 01 06:37:28.351258 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:28.349840 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079\": container with ID starting with 150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079 not found: ID does not exist" containerID="150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079" Nov 01 06:37:28.351258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.349863 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079"} err="failed to get container status \"150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079\": rpc error: code = NotFound desc = could not find container \"150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079\": container with ID starting with 150b9cc3805055c030c5841d5b0471f7dea12baf1a4bb763822fb5faa8398079 not found: ID does not exist" Nov 01 06:37:28.494068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.494017 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w"] Nov 01 06:37:28.497262 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.497244 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-df0a0131-gw-openshift-default-56c44c6f6f-zd88w"] Nov 01 06:37:28.616599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:28.616576 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" path="/var/lib/kubelet/pods/d108a036-a6a8-4a7b-ba20-aa85c4d77a52/volumes" Nov 01 06:37:29.272329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:29.272296 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:37:29.441459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:29.441429 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b"] Nov 01 06:37:31.275299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.274472 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5"] Nov 01 06:37:31.275810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.275441 4914 cpu_manager.go:414] "containerMap: removing container" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" Nov 01 06:37:31.275810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.275464 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" Nov 01 06:37:31.275810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.275631 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" Nov 01 06:37:31.275810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.275651 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" Nov 01 06:37:31.282198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.282167 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.330082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.330048 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5"] Nov 01 06:37:31.365021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.364963 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxgtd\" (UniqueName: \"kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd\") pod \"payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.365341 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.365315 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume\") pod \"payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.373943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.373918 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8b4338ad\"" Nov 01 06:37:31.374195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.374175 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8b4338ad-dockercfg-9m9qp\"" Nov 01 06:37:31.428998 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.428970 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm"] Nov 01 06:37:31.429904 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.429885 4914 cpu_manager.go:414] "containerMap: removing container" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" Nov 01 06:37:31.429951 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.429910 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d108a036-a6a8-4a7b-ba20-aa85c4d77a52" containerName="istio-proxy" Nov 01 06:37:31.434699 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.434677 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.466077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.466050 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume\") pod \"payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.466218 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.466108 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.466218 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.466186 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxgtd\" (UniqueName: \"kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd\") pod \"payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.466315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.466266 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh76t\" (UniqueName: \"kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t\") pod \"payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.466758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.466742 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume\") pod \"payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.483606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.483581 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm"] Nov 01 06:37:31.522777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.522753 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxgtd\" (UniqueName: \"kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd\") pod \"payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.567980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.567916 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.568082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.568020 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh76t\" (UniqueName: \"kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t\") pod \"payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.568573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.568553 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.615054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.615029 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:31.624807 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.624787 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh76t\" (UniqueName: \"kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t\") pod \"payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.758174 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.758148 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:31.992342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:31.992321 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5"] Nov 01 06:37:31.993134 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:37:31.993106 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a9f169_b706_4c14_9a0b_8febe8a06e75.slice/crio-8ec26fdf01aa82ae71f1b49c0f274548b4e79327f3832bd779ea3f0c09133fa7 WatchSource:0}: Error finding container 8ec26fdf01aa82ae71f1b49c0f274548b4e79327f3832bd779ea3f0c09133fa7: Status 404 returned error can't find the container with id 8ec26fdf01aa82ae71f1b49c0f274548b4e79327f3832bd779ea3f0c09133fa7 Nov 01 06:37:32.128942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.128916 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm"] Nov 01 06:37:32.129448 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:37:32.129426 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8bb5d7_13af_4a56_9e10_eeee384b247e.slice/crio-23cb76cc42c1f23a134fbd6f892598690d2d3bb1dc7d1c766efe46ed63d25e62 WatchSource:0}: Error finding container 23cb76cc42c1f23a134fbd6f892598690d2d3bb1dc7d1c766efe46ed63d25e62: Status 404 returned error can't find the container with id 23cb76cc42c1f23a134fbd6f892598690d2d3bb1dc7d1c766efe46ed63d25e62 Nov 01 06:37:32.370906 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.370877 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" event={"ID":"37a9f169-b706-4c14-9a0b-8febe8a06e75","Type":"ContainerStarted","Data":"b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8"} Nov 01 06:37:32.371323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.370924 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" event={"ID":"37a9f169-b706-4c14-9a0b-8febe8a06e75","Type":"ContainerStarted","Data":"8ec26fdf01aa82ae71f1b49c0f274548b4e79327f3832bd779ea3f0c09133fa7"} Nov 01 06:37:32.371323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.371039 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:32.372587 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.372566 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" event={"ID":"bf8bb5d7-13af-4a56-9e10-eeee384b247e","Type":"ContainerStarted","Data":"0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8"} Nov 01 06:37:32.372670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.372606 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" event={"ID":"bf8bb5d7-13af-4a56-9e10-eeee384b247e","Type":"ContainerStarted","Data":"23cb76cc42c1f23a134fbd6f892598690d2d3bb1dc7d1c766efe46ed63d25e62"} Nov 01 06:37:32.372670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.372662 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:37:32.428170 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.428128 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" podUID="16e836cc-bf7d-4391-9834-da55537b7bfd" containerName="payload-processing" containerID="cri-o://55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f" gracePeriod=25 Nov 01 06:37:32.487078 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.487038 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" podStartSLOduration=1.487026255 podStartE2EDuration="1.487026255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:32.484634354 +0000 UTC m=+1668.037423549" watchObservedRunningTime="2026-11-01 06:37:32.487026255 +0000 UTC m=+1668.039815447" Nov 01 06:37:32.609183 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.604957 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" podStartSLOduration=1.604935322 podStartE2EDuration="1.604935322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:37:32.598513424 +0000 UTC m=+1668.151302618" watchObservedRunningTime="2026-11-01 06:37:32.604935322 +0000 UTC m=+1668.157724517" Nov 01 06:37:32.791086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.791055 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:37:32.948584 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.948555 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume\") pod \"16e836cc-bf7d-4391-9834-da55537b7bfd\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " Nov 01 06:37:32.948763 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.948673 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume\") pod \"16e836cc-bf7d-4391-9834-da55537b7bfd\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " Nov 01 06:37:32.948843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.948776 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z\") pod \"16e836cc-bf7d-4391-9834-da55537b7bfd\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " Nov 01 06:37:32.948843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.948821 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials\") pod \"16e836cc-bf7d-4391-9834-da55537b7bfd\" (UID: \"16e836cc-bf7d-4391-9834-da55537b7bfd\") " Nov 01 06:37:32.948996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.948969 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume" pod "16e836cc-bf7d-4391-9834-da55537b7bfd" (UID: "16e836cc-bf7d-4391-9834-da55537b7bfd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:32.949047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.949030 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume" pod "16e836cc-bf7d-4391-9834-da55537b7bfd" (UID: "16e836cc-bf7d-4391-9834-da55537b7bfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:32.949289 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.949254 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:32.949375 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.949300 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e836cc-bf7d-4391-9834-da55537b7bfd-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:32.951131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.951106 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials" pod "16e836cc-bf7d-4391-9834-da55537b7bfd" (UID: "16e836cc-bf7d-4391-9834-da55537b7bfd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:32.951231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:32.951178 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z" pod "16e836cc-bf7d-4391-9834-da55537b7bfd" (UID: "16e836cc-bf7d-4391-9834-da55537b7bfd"). InnerVolumeSpecName "kube-api-access-2kg2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:33.051540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.051446 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kg2z\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-kube-api-access-2kg2z\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:33.051540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.051492 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/16e836cc-bf7d-4391-9834-da55537b7bfd-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:33.379588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.379556 4914 generic.go:356] "Generic (PLEG): container finished" podID="16e836cc-bf7d-4391-9834-da55537b7bfd" containerID="55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f" exitCode=0 Nov 01 06:37:33.380048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.379613 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" Nov 01 06:37:33.380048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.379650 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" event={"ID":"16e836cc-bf7d-4391-9834-da55537b7bfd","Type":"ContainerDied","Data":"55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f"} Nov 01 06:37:33.380048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.379705 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l" event={"ID":"16e836cc-bf7d-4391-9834-da55537b7bfd","Type":"ContainerDied","Data":"92324680dfdf440535e60796943cba62e6ed9310b6d373ccef323d3d11b5de19"} Nov 01 06:37:33.380048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.379719 4914 scope.go:122] "RemoveContainer" containerID="55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f" Nov 01 06:37:33.392343 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.392322 4914 scope.go:122] "RemoveContainer" containerID="55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f" Nov 01 06:37:33.392596 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:33.392579 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f\": container with ID starting with 55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f not found: ID does not exist" containerID="55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f" Nov 01 06:37:33.392650 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.392605 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f"} err="failed to get container status \"55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f\": rpc error: code = NotFound desc = could not find container \"55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f\": container with ID starting with 55605f4be7a8cac53f7e9d16bce22ab1bbfbc75b34786e20debe80e50a2c119f not found: ID does not exist" Nov 01 06:37:33.543780 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.543759 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l"] Nov 01 06:37:33.546286 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:33.546254 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6lrw6l"] Nov 01 06:37:34.442408 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.442378 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" podUID="1792c887-eb33-4b49-8c73-8fd056b3d3e5" containerName="payload-processing" containerID="cri-o://20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef" gracePeriod=25 Nov 01 06:37:34.624399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.621114 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e836cc-bf7d-4391-9834-da55537b7bfd" path="/var/lib/kubelet/pods/16e836cc-bf7d-4391-9834-da55537b7bfd/volumes" Nov 01 06:37:34.698129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.698077 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:37:34.867725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.867699 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials\") pod \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " Nov 01 06:37:34.867885 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.867783 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4\") pod \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " Nov 01 06:37:34.867949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.867883 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume\") pod \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " Nov 01 06:37:34.867949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.867920 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume\") pod \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\" (UID: \"1792c887-eb33-4b49-8c73-8fd056b3d3e5\") " Nov 01 06:37:34.868296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.868244 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume" pod "1792c887-eb33-4b49-8c73-8fd056b3d3e5" (UID: "1792c887-eb33-4b49-8c73-8fd056b3d3e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:34.868573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.868337 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:34.868573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.868382 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume" pod "1792c887-eb33-4b49-8c73-8fd056b3d3e5" (UID: "1792c887-eb33-4b49-8c73-8fd056b3d3e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:37:34.870114 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.870095 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials" pod "1792c887-eb33-4b49-8c73-8fd056b3d3e5" (UID: "1792c887-eb33-4b49-8c73-8fd056b3d3e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:34.870205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.870149 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4" pod "1792c887-eb33-4b49-8c73-8fd056b3d3e5" (UID: "1792c887-eb33-4b49-8c73-8fd056b3d3e5"). InnerVolumeSpecName "kube-api-access-j5gb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:37:34.969678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.969614 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1792c887-eb33-4b49-8c73-8fd056b3d3e5-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:34.969678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.969639 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:34.969678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:34.969648 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gb4\" (UniqueName: \"kubernetes.io/projected/1792c887-eb33-4b49-8c73-8fd056b3d3e5-kube-api-access-j5gb4\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:37:35.393841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.393810 4914 generic.go:356] "Generic (PLEG): container finished" podID="1792c887-eb33-4b49-8c73-8fd056b3d3e5" containerID="20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef" exitCode=0 Nov 01 06:37:35.393997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.393859 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" Nov 01 06:37:35.393997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.393869 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" event={"ID":"1792c887-eb33-4b49-8c73-8fd056b3d3e5","Type":"ContainerDied","Data":"20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef"} Nov 01 06:37:35.393997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.393910 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b" event={"ID":"1792c887-eb33-4b49-8c73-8fd056b3d3e5","Type":"ContainerDied","Data":"6d92dc0fcd40b5a67eaf9ea6f6b2973b4772ba5f3e5a9e02f514085403ceb405"} Nov 01 06:37:35.393997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.393930 4914 scope.go:122] "RemoveContainer" containerID="20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef" Nov 01 06:37:35.403683 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.403668 4914 scope.go:122] "RemoveContainer" containerID="20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef" Nov 01 06:37:35.403927 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:35.403908 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef\": container with ID starting with 20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef not found: ID does not exist" containerID="20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef" Nov 01 06:37:35.403977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.403935 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef"} err="failed to get container status \"20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef\": rpc error: code = NotFound desc = could not find container \"20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef\": container with ID starting with 20a12c7612a423bcdfb0adb64d55c4085938dcf109633284516269ee0f0416ef not found: ID does not exist" Nov 01 06:37:35.557996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.557973 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b"] Nov 01 06:37:35.607926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:35.607902 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79fmzn7b"] Nov 01 06:37:36.615844 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:36.615814 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1792c887-eb33-4b49-8c73-8fd056b3d3e5" path="/var/lib/kubelet/pods/1792c887-eb33-4b49-8c73-8fd056b3d3e5/volumes" Nov 01 06:37:37.592059 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:37:37.592023 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:37:43.382668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:43.382636 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:37:43.383163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:37:43.382844 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:38:07.591500 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:38:07.591467 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:38:11.115838 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.115806 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m"] Nov 01 06:38:11.116438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.116423 4914 cpu_manager.go:414] "containerMap: removing container" podUID="16e836cc-bf7d-4391-9834-da55537b7bfd" containerName="payload-processing" Nov 01 06:38:11.116490 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.116439 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e836cc-bf7d-4391-9834-da55537b7bfd" containerName="payload-processing" Nov 01 06:38:11.116524 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.116493 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1792c887-eb33-4b49-8c73-8fd056b3d3e5" containerName="payload-processing" Nov 01 06:38:11.116524 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.116499 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1792c887-eb33-4b49-8c73-8fd056b3d3e5" containerName="payload-processing" Nov 01 06:38:11.116594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.116584 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1792c887-eb33-4b49-8c73-8fd056b3d3e5" containerName="payload-processing" Nov 01 06:38:11.116633 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.116598 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e836cc-bf7d-4391-9834-da55537b7bfd" containerName="payload-processing" Nov 01 06:38:11.120189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.120169 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.223494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.223462 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8b4338ad-serving-cert\"" Nov 01 06:38:11.226322 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.226300 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m"] Nov 01 06:38:11.258099 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.258072 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.258183 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.258108 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.258183 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.258132 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs94x\" (UniqueName: \"kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.258307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.258256 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.359048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.359014 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.359195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.359150 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.359195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.359191 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.359324 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:38:11.359305 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-serving-cert: secret "maas-api-e2e-ait-8b4338ad-serving-cert" not found Nov 01 06:38:11.359381 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.359316 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs94x\" (UniqueName: \"kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.359381 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:38:11.359369 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls podName:487bae86-68ff-41d3-a61f-9414f8497a7b nodeName:}" failed. No retries permitted until 2026-11-01 06:38:11.859345667 +0000 UTC m=+1707.412134839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls") pod "maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" (UID: "487bae86-68ff-41d3-a61f-9414f8497a7b") : secret "maas-api-e2e-ait-8b4338ad-serving-cert" not found Nov 01 06:38:11.359732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.359709 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.361745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.361725 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.443126 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.443034 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs94x\" (UniqueName: \"kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.867639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.867606 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:11.870133 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:11.870108 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\") pod \"maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:12.033633 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:12.033608 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:12.406212 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:12.406187 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m"] Nov 01 06:38:12.406682 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:38:12.406662 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487bae86_68ff_41d3_a61f_9414f8497a7b.slice/crio-fecc2310f10692c35d9e3dbf68a37eb6e12d8eec9e135f9e8e452c21b878ead5 WatchSource:0}: Error finding container fecc2310f10692c35d9e3dbf68a37eb6e12d8eec9e135f9e8e452c21b878ead5: Status 404 returned error can't find the container with id fecc2310f10692c35d9e3dbf68a37eb6e12d8eec9e135f9e8e452c21b878ead5 Nov 01 06:38:12.408136 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:12.408115 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:38:12.667578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:12.667512 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" event={"ID":"487bae86-68ff-41d3-a61f-9414f8497a7b","Type":"ContainerStarted","Data":"fecc2310f10692c35d9e3dbf68a37eb6e12d8eec9e135f9e8e452c21b878ead5"} Nov 01 06:38:13.674407 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:13.674369 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" event={"ID":"487bae86-68ff-41d3-a61f-9414f8497a7b","Type":"ContainerStarted","Data":"199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1"} Nov 01 06:38:13.674865 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:13.674495 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:13.793895 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:13.793850 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" podStartSLOduration=1.917024691 podStartE2EDuration="2.793837083s" totalImagesPullingTime="876.812392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:13.792355377 +0000 UTC m=+1709.345144572" watchObservedRunningTime="2026-11-01 06:38:13.793837083 +0000 UTC m=+1709.346626330" Nov 01 06:38:14.946105 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:14.946074 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg"] Nov 01 06:38:14.962537 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:14.962507 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.007853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.007802 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg"] Nov 01 06:38:15.143308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.141538 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-51aa8f-dockercfg-49tvx\"" Nov 01 06:38:15.143308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.143269 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-51aa8f\"" Nov 01 06:38:15.161302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.151853 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct"] Nov 01 06:38:15.172317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.171207 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct"] Nov 01 06:38:15.172317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.171338 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.256187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.256118 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.256332 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.256245 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcz2\" (UniqueName: \"kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2\") pod \"payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.357077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.357051 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.357227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.357102 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.357227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.357148 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcz2\" (UniqueName: \"kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2\") pod \"payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.357369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.357258 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fxr5\" (UniqueName: \"kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5\") pod \"payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.357873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.357850 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.414109 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.414090 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcz2\" (UniqueName: \"kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2\") pod \"payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.458018 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.457996 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fxr5\" (UniqueName: \"kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5\") pod \"payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.458118 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.458070 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.458670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.458653 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.514661 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.514609 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fxr5\" (UniqueName: \"kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5\") pod \"payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.576044 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.576023 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:15.787459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.787385 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:15.972363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:15.972334 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg"] Nov 01 06:38:15.972692 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:38:15.972416 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90e61ce_cc69_4f71_b18a_23b8da40aeab.slice/crio-8d6908c6bfa003d91bd11762fe15cb1ad3a7163d8822fbb3eef770030bd93ee3 WatchSource:0}: Error finding container 8d6908c6bfa003d91bd11762fe15cb1ad3a7163d8822fbb3eef770030bd93ee3: Status 404 returned error can't find the container with id 8d6908c6bfa003d91bd11762fe15cb1ad3a7163d8822fbb3eef770030bd93ee3 Nov 01 06:38:16.174260 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:38:16.174234 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f27112_de78_4352_bee3_b1533e827e0d.slice/crio-6c44d8242199bb11f7d578d41d5b29ab3a5dcab8d89e5c506f20647b7852e054 WatchSource:0}: Error finding container 6c44d8242199bb11f7d578d41d5b29ab3a5dcab8d89e5c506f20647b7852e054: Status 404 returned error can't find the container with id 6c44d8242199bb11f7d578d41d5b29ab3a5dcab8d89e5c506f20647b7852e054 Nov 01 06:38:16.176439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.176417 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct"] Nov 01 06:38:16.768082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.768044 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" event={"ID":"b4f27112-de78-4352-bee3-b1533e827e0d","Type":"ContainerStarted","Data":"0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99"} Nov 01 06:38:16.768318 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.768298 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:16.768432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.768344 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" event={"ID":"b4f27112-de78-4352-bee3-b1533e827e0d","Type":"ContainerStarted","Data":"6c44d8242199bb11f7d578d41d5b29ab3a5dcab8d89e5c506f20647b7852e054"} Nov 01 06:38:16.769849 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.769820 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" event={"ID":"f90e61ce-cc69-4f71-b18a-23b8da40aeab","Type":"ContainerStarted","Data":"1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de"} Nov 01 06:38:16.769973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.769856 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" event={"ID":"f90e61ce-cc69-4f71-b18a-23b8da40aeab","Type":"ContainerStarted","Data":"8d6908c6bfa003d91bd11762fe15cb1ad3a7163d8822fbb3eef770030bd93ee3"} Nov 01 06:38:16.769973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.769907 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:16.885704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:16.885654 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" podStartSLOduration=1.885639952 podStartE2EDuration="1.885639952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:16.88303614 +0000 UTC m=+1712.435825331" watchObservedRunningTime="2026-11-01 06:38:16.885639952 +0000 UTC m=+1712.438429179" Nov 01 06:38:17.026255 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:17.026099 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" podStartSLOduration=3.026067607 podStartE2EDuration="3.026067607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:38:17.006400694 +0000 UTC m=+1712.559189886" watchObservedRunningTime="2026-11-01 06:38:17.026067607 +0000 UTC m=+1712.578856800" Nov 01 06:38:19.752219 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:19.752192 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:38:27.778158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:27.778131 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 06:38:27.778626 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:27.778191 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 06:38:37.595680 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:38:37.595640 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:38:42.099589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.099558 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": read tcp 10.132.0.2:46838->10.132.0.96:15021: read: connection reset by peer" start-of-body= Nov 01 06:38:42.100006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.099623 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": read tcp 10.132.0.2:46838->10.132.0.96:15021: read: connection reset by peer" Nov 01 06:38:42.932103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.932070 4914 generic.go:356] "Generic (PLEG): container finished" podID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerID="8bd3b58a90cbfc2b526a1867bc09dc3cc7c3722813aad81728a71959074cf8d1" exitCode=0 Nov 01 06:38:42.932263 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.932143 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" event={"ID":"5e8304a7-c449-48f1-9545-ac3e732eba47","Type":"ContainerDied","Data":"8bd3b58a90cbfc2b526a1867bc09dc3cc7c3722813aad81728a71959074cf8d1"} Nov 01 06:38:42.932466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.932453 4914 scope.go:122] "RemoveContainer" containerID="8bd3b58a90cbfc2b526a1867bc09dc3cc7c3722813aad81728a71959074cf8d1" Nov 01 06:38:42.933648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.933619 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:38:42.933716 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.933690 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:38:42.933758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:42.933720 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:38:43.939839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:43.939805 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" event={"ID":"5e8304a7-c449-48f1-9545-ac3e732eba47","Type":"ContainerStarted","Data":"f89fe2a92e8a6a0edbd7cd114808b768b72ce0d801a40e450071be00b6570015"} Nov 01 06:38:44.242990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:44.242914 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:38:44.243130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:44.243054 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 06:38:44.243130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:44.243113 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 06:38:45.242517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:45.242482 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 06:38:45.242979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:45.242532 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 06:38:46.242557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:46.242525 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 06:38:46.242925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:46.242575 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 06:38:47.242476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:47.242437 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 06:38:47.242652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:47.242494 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 06:38:48.242890 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:48.242857 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 01 06:38:48.243305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:48.242915 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 01 06:38:49.246683 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:49.246655 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:38:49.247067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:49.247048 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:38:49.247808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:49.247783 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:38:58.658900 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.658827 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn"] Nov 01 06:38:58.662326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.662307 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.719475 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.719448 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.719583 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.719507 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.719732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.719704 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd97\" (UniqueName: \"kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.719991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.719969 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.766507 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.766481 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-51aa8f-serving-cert\"" Nov 01 06:38:58.769549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.769526 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn"] Nov 01 06:38:58.821269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.821242 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.821435 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.821354 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.821435 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:38:58.821397 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-serving-cert: secret "maas-api-e2e-shared-b-w3-51aa8f-serving-cert" not found Nov 01 06:38:58.821543 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:38:58.821472 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls podName:edcdd90e-cdf8-4c72-a634-a50cd15e9cbf nodeName:}" failed. No retries permitted until 2026-11-01 06:38:59.321453084 +0000 UTC m=+1754.874242257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls") pod "maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" (UID: "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf") : secret "maas-api-e2e-shared-b-w3-51aa8f-serving-cert" not found Nov 01 06:38:58.821543 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.821403 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.821656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.821580 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd97\" (UniqueName: \"kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.822004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.821977 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.823862 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.823833 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:58.878848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:58.878826 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd97\" (UniqueName: \"kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:59.329659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:59.329628 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:59.332081 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:59.332060 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:59.573779 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:59.573754 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:38:59.944146 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:38:59.944115 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn"] Nov 01 06:38:59.945016 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:38:59.944994 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcdd90e_cdf8_4c72_a634_a50cd15e9cbf.slice/crio-c6d166ee4b79aa5dd512f2b34f54d359887313c116d7d99dee1b559157975dfa WatchSource:0}: Error finding container c6d166ee4b79aa5dd512f2b34f54d359887313c116d7d99dee1b559157975dfa: Status 404 returned error can't find the container with id c6d166ee4b79aa5dd512f2b34f54d359887313c116d7d99dee1b559157975dfa Nov 01 06:39:00.063245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:00.063217 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" event={"ID":"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf","Type":"ContainerStarted","Data":"c6d166ee4b79aa5dd512f2b34f54d359887313c116d7d99dee1b559157975dfa"} Nov 01 06:39:01.070156 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:01.070076 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" event={"ID":"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf","Type":"ContainerStarted","Data":"d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2"} Nov 01 06:39:01.070585 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:01.070178 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:39:01.186230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:01.186172 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" podStartSLOduration=2.372492862 podStartE2EDuration="3.186160566s" totalImagesPullingTime="813.667704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:39:01.183766639 +0000 UTC m=+1756.736555830" watchObservedRunningTime="2026-11-01 06:39:01.186160566 +0000 UTC m=+1756.738949786" Nov 01 06:39:07.081412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:07.081380 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 06:39:07.591786 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:39:07.591752 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:39:22.272382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:22.272344 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5"] Nov 01 06:39:22.383816 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:22.383785 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm"] Nov 01 06:39:27.273620 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.273581 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" podUID="37a9f169-b706-4c14-9a0b-8febe8a06e75" containerName="payload-processing" containerID="cri-o://b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8" gracePeriod=25 Nov 01 06:39:27.384675 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.384643 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" podUID="bf8bb5d7-13af-4a56-9e10-eeee384b247e" containerName="payload-pre-processing" containerID="cri-o://0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8" gracePeriod=25 Nov 01 06:39:27.539553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.539491 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:39:27.652738 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.652715 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:39:27.679104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.679074 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume\") pod \"37a9f169-b706-4c14-9a0b-8febe8a06e75\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " Nov 01 06:39:27.679249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.679158 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd\" (UniqueName: \"kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd\") pod \"37a9f169-b706-4c14-9a0b-8febe8a06e75\" (UID: \"37a9f169-b706-4c14-9a0b-8febe8a06e75\") " Nov 01 06:39:27.679472 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.679450 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume" pod "37a9f169-b706-4c14-9a0b-8febe8a06e75" (UID: "37a9f169-b706-4c14-9a0b-8febe8a06e75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:39:27.679863 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.679841 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a9f169-b706-4c14-9a0b-8febe8a06e75-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:27.681257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.681235 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd" pod "37a9f169-b706-4c14-9a0b-8febe8a06e75" (UID: "37a9f169-b706-4c14-9a0b-8febe8a06e75"). InnerVolumeSpecName "kube-api-access-vxgtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:39:27.780632 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.780607 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume\") pod \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " Nov 01 06:39:27.780800 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.780785 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t\" (UniqueName: \"kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t\") pod \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\" (UID: \"bf8bb5d7-13af-4a56-9e10-eeee384b247e\") " Nov 01 06:39:27.781056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.781030 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume" pod "bf8bb5d7-13af-4a56-9e10-eeee384b247e" (UID: "bf8bb5d7-13af-4a56-9e10-eeee384b247e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:39:27.781260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.781238 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxgtd\" (UniqueName: \"kubernetes.io/projected/37a9f169-b706-4c14-9a0b-8febe8a06e75-kube-api-access-vxgtd\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:27.781374 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.781265 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf8bb5d7-13af-4a56-9e10-eeee384b247e-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:27.783129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.783106 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t" pod "bf8bb5d7-13af-4a56-9e10-eeee384b247e" (UID: "bf8bb5d7-13af-4a56-9e10-eeee384b247e"). InnerVolumeSpecName "kube-api-access-dh76t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:39:27.881993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:27.881965 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh76t\" (UniqueName: \"kubernetes.io/projected/bf8bb5d7-13af-4a56-9e10-eeee384b247e-kube-api-access-dh76t\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:28.236613 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.236540 4914 generic.go:356] "Generic (PLEG): container finished" podID="37a9f169-b706-4c14-9a0b-8febe8a06e75" containerID="b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8" exitCode=0 Nov 01 06:39:28.236736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.236642 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" event={"ID":"37a9f169-b706-4c14-9a0b-8febe8a06e75","Type":"ContainerDied","Data":"b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8"} Nov 01 06:39:28.236791 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.236754 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" Nov 01 06:39:28.236831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.236805 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5" event={"ID":"37a9f169-b706-4c14-9a0b-8febe8a06e75","Type":"ContainerDied","Data":"8ec26fdf01aa82ae71f1b49c0f274548b4e79327f3832bd779ea3f0c09133fa7"} Nov 01 06:39:28.236898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.236841 4914 scope.go:122] "RemoveContainer" containerID="b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8" Nov 01 06:39:28.238206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.238181 4914 generic.go:356] "Generic (PLEG): container finished" podID="bf8bb5d7-13af-4a56-9e10-eeee384b247e" containerID="0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8" exitCode=0 Nov 01 06:39:28.238327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.238231 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" Nov 01 06:39:28.238327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.238295 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" event={"ID":"bf8bb5d7-13af-4a56-9e10-eeee384b247e","Type":"ContainerDied","Data":"0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8"} Nov 01 06:39:28.238598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.238338 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm" event={"ID":"bf8bb5d7-13af-4a56-9e10-eeee384b247e","Type":"ContainerDied","Data":"23cb76cc42c1f23a134fbd6f892598690d2d3bb1dc7d1c766efe46ed63d25e62"} Nov 01 06:39:28.249552 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.249530 4914 scope.go:122] "RemoveContainer" containerID="b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8" Nov 01 06:39:28.250088 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:39:28.250054 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8\": container with ID starting with b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8 not found: ID does not exist" containerID="b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8" Nov 01 06:39:28.250181 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.250097 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8"} err="failed to get container status \"b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8\": rpc error: code = NotFound desc = could not find container \"b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8\": container with ID starting with b72dbacbab32141f6494622e80100b4054c9f5f7da9eb233c344dca7dae437f8 not found: ID does not exist" Nov 01 06:39:28.250181 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.250113 4914 scope.go:122] "RemoveContainer" containerID="0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8" Nov 01 06:39:28.258821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.258808 4914 scope.go:122] "RemoveContainer" containerID="0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8" Nov 01 06:39:28.259062 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:39:28.259044 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8\": container with ID starting with 0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8 not found: ID does not exist" containerID="0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8" Nov 01 06:39:28.259098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.259069 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8"} err="failed to get container status \"0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8\": rpc error: code = NotFound desc = could not find container \"0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8\": container with ID starting with 0cf2fde21b53f486a5fd6742e0af32102f607118f8809c770755b959780a3db8 not found: ID does not exist" Nov 01 06:39:28.402133 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.402109 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5"] Nov 01 06:39:28.451983 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.451950 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8b4338ad-669b6c55cc-xjtz5"] Nov 01 06:39:28.513665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.513605 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm"] Nov 01 06:39:28.516047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.516029 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8b4338ad-659d674775-jvhnm"] Nov 01 06:39:28.616653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.616633 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a9f169-b706-4c14-9a0b-8febe8a06e75" path="/var/lib/kubelet/pods/37a9f169-b706-4c14-9a0b-8febe8a06e75/volumes" Nov 01 06:39:28.617021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:28.617008 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8bb5d7-13af-4a56-9e10-eeee384b247e" path="/var/lib/kubelet/pods/bf8bb5d7-13af-4a56-9e10-eeee384b247e/volumes" Nov 01 06:39:37.591929 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:39:37.591894 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:39:44.818926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:44.818899 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:39:44.824860 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:44.824832 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:39:56.578443 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.577106 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m"] Nov 01 06:39:56.578443 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.577717 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" podUID="487bae86-68ff-41d3-a61f-9414f8497a7b" containerName="maas-api" containerID="cri-o://199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1" gracePeriod=30 Nov 01 06:39:56.914583 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.914561 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:39:56.969266 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.969238 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca\") pod \"487bae86-68ff-41d3-a61f-9414f8497a7b\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " Nov 01 06:39:56.969393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.969337 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\") pod \"487bae86-68ff-41d3-a61f-9414f8497a7b\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " Nov 01 06:39:56.969393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.969383 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x\" (UniqueName: \"kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x\") pod \"487bae86-68ff-41d3-a61f-9414f8497a7b\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " Nov 01 06:39:56.969504 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.969417 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs\") pod \"487bae86-68ff-41d3-a61f-9414f8497a7b\" (UID: \"487bae86-68ff-41d3-a61f-9414f8497a7b\") " Nov 01 06:39:56.969602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.969576 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca" pod "487bae86-68ff-41d3-a61f-9414f8497a7b" (UID: "487bae86-68ff-41d3-a61f-9414f8497a7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:39:56.969808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.969788 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487bae86-68ff-41d3-a61f-9414f8497a7b-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:56.971498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.971465 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls" pod "487bae86-68ff-41d3-a61f-9414f8497a7b" (UID: "487bae86-68ff-41d3-a61f-9414f8497a7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:39:56.971598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.971517 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x" pod "487bae86-68ff-41d3-a61f-9414f8497a7b" (UID: "487bae86-68ff-41d3-a61f-9414f8497a7b"). InnerVolumeSpecName "kube-api-access-bs94x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:39:56.971598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:56.971519 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs" pod "487bae86-68ff-41d3-a61f-9414f8497a7b" (UID: "487bae86-68ff-41d3-a61f-9414f8497a7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:39:57.071038 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.071005 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:57.071038 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.071036 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs94x\" (UniqueName: \"kubernetes.io/projected/487bae86-68ff-41d3-a61f-9414f8497a7b-kube-api-access-bs94x\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:57.071224 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.071048 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487bae86-68ff-41d3-a61f-9414f8497a7b-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:39:57.486294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.480945 4914 generic.go:356] "Generic (PLEG): container finished" podID="487bae86-68ff-41d3-a61f-9414f8497a7b" containerID="199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1" exitCode=0 Nov 01 06:39:57.486294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.481126 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" event={"ID":"487bae86-68ff-41d3-a61f-9414f8497a7b","Type":"ContainerDied","Data":"199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1"} Nov 01 06:39:57.486294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.481212 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" event={"ID":"487bae86-68ff-41d3-a61f-9414f8497a7b","Type":"ContainerDied","Data":"fecc2310f10692c35d9e3dbf68a37eb6e12d8eec9e135f9e8e452c21b878ead5"} Nov 01 06:39:57.486294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.481241 4914 scope.go:122] "RemoveContainer" containerID="199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1" Nov 01 06:39:57.486294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.481447 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m" Nov 01 06:39:57.535639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.535607 4914 scope.go:122] "RemoveContainer" containerID="199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1" Nov 01 06:39:57.535933 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:39:57.535915 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1\": container with ID starting with 199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1 not found: ID does not exist" containerID="199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1" Nov 01 06:39:57.536023 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.535941 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1"} err="failed to get container status \"199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1\": rpc error: code = NotFound desc = could not find container \"199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1\": container with ID starting with 199b9e06d3f2264f961650c738846481e3909bd4c0de1e6bf7b5c740569518d1 not found: ID does not exist" Nov 01 06:39:57.677879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.677854 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m"] Nov 01 06:39:57.727968 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:57.727935 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8b4338ad-6b5ffbdcd9-9cq5m"] Nov 01 06:39:58.616347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:39:58.616315 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487bae86-68ff-41d3-a61f-9414f8497a7b" path="/var/lib/kubelet/pods/487bae86-68ff-41d3-a61f-9414f8497a7b/volumes" Nov 01 06:40:00.309645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:00.309618 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz"] Nov 01 06:40:00.310069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:00.309813 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" containerID="cri-o://f89fe2a92e8a6a0edbd7cd114808b768b72ce0d801a40e450071be00b6570015" gracePeriod=30 Nov 01 06:40:01.427835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.427797 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs"] Nov 01 06:40:01.428535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428515 4914 cpu_manager.go:414] "containerMap: removing container" podUID="bf8bb5d7-13af-4a56-9e10-eeee384b247e" containerName="payload-pre-processing" Nov 01 06:40:01.428535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428536 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8bb5d7-13af-4a56-9e10-eeee384b247e" containerName="payload-pre-processing" Nov 01 06:40:01.428688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428572 4914 cpu_manager.go:414] "containerMap: removing container" podUID="37a9f169-b706-4c14-9a0b-8febe8a06e75" containerName="payload-processing" Nov 01 06:40:01.428688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428578 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a9f169-b706-4c14-9a0b-8febe8a06e75" containerName="payload-processing" Nov 01 06:40:01.428688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428595 4914 cpu_manager.go:414] "containerMap: removing container" podUID="487bae86-68ff-41d3-a61f-9414f8497a7b" containerName="maas-api" Nov 01 06:40:01.428688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428600 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487bae86-68ff-41d3-a61f-9414f8497a7b" containerName="maas-api" Nov 01 06:40:01.428688 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428678 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="487bae86-68ff-41d3-a61f-9414f8497a7b" containerName="maas-api" Nov 01 06:40:01.428941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428693 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8bb5d7-13af-4a56-9e10-eeee384b247e" containerName="payload-pre-processing" Nov 01 06:40:01.428941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.428701 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a9f169-b706-4c14-9a0b-8febe8a06e75" containerName="payload-processing" Nov 01 06:40:01.436307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.436259 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.528768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.528735 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4c66acdd-openshift-default-dockercfg-xtbqj\"" Nov 01 06:40:01.570811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.570774 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs"] Nov 01 06:40:01.601997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.601955 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602018 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nbkj\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602063 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602164 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602157 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602325 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602240 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602325 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602255 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602338 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602587 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.602841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.602816 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704089 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704008 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704089 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704044 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nbkj\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704201 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704354 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704468 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704436 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704571 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704550 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704633 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704584 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704684 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704660 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704725 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704790 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704778 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704920 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704897 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.704999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.704979 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.705058 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.705022 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.705113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.705096 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.707026 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.707005 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.707131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.707086 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.760976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.760952 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.761075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.760977 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nbkj\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj\") pod \"e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:01.822257 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:01.822239 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:02.270919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.270893 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs"] Nov 01 06:40:02.271921 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:40:02.271895 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda714f6bb_9938_46d3_8ee1_dab54ec89a84.slice/crio-9ea9693a9504f333a4fbba06f4da3b1dda48e7cabde74747fe7107c3af8e2fd0 WatchSource:0}: Error finding container 9ea9693a9504f333a4fbba06f4da3b1dda48e7cabde74747fe7107c3af8e2fd0: Status 404 returned error can't find the container with id 9ea9693a9504f333a4fbba06f4da3b1dda48e7cabde74747fe7107c3af8e2fd0 Nov 01 06:40:02.274158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.274125 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:40:02.274251 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.274195 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:40:02.274351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.274295 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:40:02.535526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.535447 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" event={"ID":"a714f6bb-9938-46d3-8ee1-dab54ec89a84","Type":"ContainerStarted","Data":"d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172"} Nov 01 06:40:02.535526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.535501 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" event={"ID":"a714f6bb-9938-46d3-8ee1-dab54ec89a84","Type":"ContainerStarted","Data":"9ea9693a9504f333a4fbba06f4da3b1dda48e7cabde74747fe7107c3af8e2fd0"} Nov 01 06:40:02.659913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.659853 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podStartSLOduration=1.659840274 podStartE2EDuration="1.659840274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:02.657162166 +0000 UTC m=+1818.209951360" watchObservedRunningTime="2026-11-01 06:40:02.659840274 +0000 UTC m=+1818.212629463" Nov 01 06:40:02.822588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:02.822401 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:03.833168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:03.833131 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:04.249478 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:04.249444 4914 patch_prober.go:29] interesting pod/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:40:04.249653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:04.249500 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:40:04.547403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:04.547342 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:04.548100 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:04.548081 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:40:05.556374 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.556346 4914 generic.go:356] "Generic (PLEG): container finished" podID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerID="f89fe2a92e8a6a0edbd7cd114808b768b72ce0d801a40e450071be00b6570015" exitCode=0 Nov 01 06:40:05.556797 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.556395 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" event={"ID":"5e8304a7-c449-48f1-9545-ac3e732eba47","Type":"ContainerDied","Data":"f89fe2a92e8a6a0edbd7cd114808b768b72ce0d801a40e450071be00b6570015"} Nov 01 06:40:05.556797 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.556464 4914 scope.go:122] "RemoveContainer" containerID="8bd3b58a90cbfc2b526a1867bc09dc3cc7c3722813aad81728a71959074cf8d1" Nov 01 06:40:05.596746 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.596722 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:40:05.650440 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.650420 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.650653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.650629 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:05.650753 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.650738 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.751000 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.750975 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751005 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751038 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751065 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751119 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751204 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751259 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751303 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data\") pod \"5e8304a7-c449-48f1-9545-ac3e732eba47\" (UID: \"5e8304a7-c449-48f1-9545-ac3e732eba47\") " Nov 01 06:40:05.751351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751325 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:05.751548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.751419 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:05.752189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.752156 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:05.752189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.752180 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:05.753316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.753290 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8304a7-c449-48f1-9545-ac3e732eba47-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.753316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.753316 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.753479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.753340 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.753479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.753368 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.753645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.753614 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:05.753953 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.753935 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "kube-api-access-6cr5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:05.756422 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.756392 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:05.764551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.764508 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token" pod "5e8304a7-c449-48f1-9545-ac3e732eba47" (UID: "5e8304a7-c449-48f1-9545-ac3e732eba47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:05.854880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.854855 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.854880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.854880 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.855069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.854889 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:05.855069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:05.854897 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cr5m\" (UniqueName: \"kubernetes.io/projected/5e8304a7-c449-48f1-9545-ac3e732eba47-kube-api-access-6cr5m\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:06.456582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.456550 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv"] Nov 01 06:40:06.457198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.457183 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" Nov 01 06:40:06.457249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.457198 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" Nov 01 06:40:06.457249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.457214 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" Nov 01 06:40:06.457249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.457218 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" Nov 01 06:40:06.457396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.457341 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" Nov 01 06:40:06.457396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.457350 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" containerName="istio-proxy" Nov 01 06:40:06.461012 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.460993 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.539902 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.539873 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv"] Nov 01 06:40:06.562165 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.562141 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.562570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.562188 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.562570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.562253 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.562570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.562410 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.562570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.562455 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.562570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.562480 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngpr\" (UniqueName: \"kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.563739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.563720 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" Nov 01 06:40:06.563858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.563735 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz" event={"ID":"5e8304a7-c449-48f1-9545-ac3e732eba47","Type":"ContainerDied","Data":"f58098317a40ef70db4b63500880736317c7187825bc1d7640854529251287e9"} Nov 01 06:40:06.563858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.563802 4914 scope.go:122] "RemoveContainer" containerID="f89fe2a92e8a6a0edbd7cd114808b768b72ce0d801a40e450071be00b6570015" Nov 01 06:40:06.563972 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.563870 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2e0116\"/\"openshift-service-ca.crt\"" Nov 01 06:40:06.563972 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.563884 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2e0116\"/\"kube-root-ca.crt\"" Nov 01 06:40:06.564080 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.564038 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2e0116\"/\"default-dockercfg-2kcl4\"" Nov 01 06:40:06.564122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.564088 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2e0116\"/\"auth-test-model-2e0116-kserve-self-signed-certs\"" Nov 01 06:40:06.664406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.664363 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.665103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665075 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.665265 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665248 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.665392 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665358 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.665392 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665384 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngpr\" (UniqueName: \"kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.665518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665501 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.665859 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665837 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.667014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.666207 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.667153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.665020 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.669724 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.669699 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.670668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.670641 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.705884 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.705861 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz"] Nov 01 06:40:06.709598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.709547 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8b4338ad-openshift-default-57f75f84b7-9nhzz"] Nov 01 06:40:06.726854 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.726821 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngpr\" (UniqueName: \"kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr\") pod \"auth-test-model-2e0116-kserve-7db55d86d5-ttdtv\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:06.774366 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:06.774346 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:07.143391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:07.143364 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv"] Nov 01 06:40:07.144146 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:40:07.144121 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaff97354_8e10_41db_b61e_b2828b6dc0c0.slice/crio-e3092a4f93ee20329bd5a8f01e1f3d86a29b340de875b96c3b2fe464cd2b70ae WatchSource:0}: Error finding container e3092a4f93ee20329bd5a8f01e1f3d86a29b340de875b96c3b2fe464cd2b70ae: Status 404 returned error can't find the container with id e3092a4f93ee20329bd5a8f01e1f3d86a29b340de875b96c3b2fe464cd2b70ae Nov 01 06:40:07.571589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:07.571509 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" event={"ID":"aff97354-8e10-41db-b61e-b2828b6dc0c0","Type":"ContainerStarted","Data":"048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2"} Nov 01 06:40:07.571589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:07.571553 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" event={"ID":"aff97354-8e10-41db-b61e-b2828b6dc0c0","Type":"ContainerStarted","Data":"e3092a4f93ee20329bd5a8f01e1f3d86a29b340de875b96c3b2fe464cd2b70ae"} Nov 01 06:40:07.593248 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:40:07.593212 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:40:07.686683 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:07.686634 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" podStartSLOduration=1.686622346 podStartE2EDuration="1.686622346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:07.684600492 +0000 UTC m=+1823.237389683" watchObservedRunningTime="2026-11-01 06:40:07.686622346 +0000 UTC m=+1823.239411537" Nov 01 06:40:07.775315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:07.775270 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:08.616142 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:08.616115 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8304a7-c449-48f1-9545-ac3e732eba47" path="/var/lib/kubelet/pods/5e8304a7-c449-48f1-9545-ac3e732eba47/volumes" Nov 01 06:40:11.432460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.432421 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh"] Nov 01 06:40:11.437823 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.437797 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.492680 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.492654 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh"] Nov 01 06:40:11.586300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.561648 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c66acdd\"" Nov 01 06:40:11.586300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.562375 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c66acdd-dockercfg-s6qjs\"" Nov 01 06:40:11.607667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.607529 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf"] Nov 01 06:40:11.616304 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.616247 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.625567 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.625540 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dhp\" (UniqueName: \"kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp\") pod \"payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.625802 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.618871 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.667930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.667898 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf"] Nov 01 06:40:11.727154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.727080 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxtv\" (UniqueName: \"kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv\") pod \"payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.727154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.727128 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.727347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.727267 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.727347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.727322 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dhp\" (UniqueName: \"kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp\") pod \"payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.727906 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.727885 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.784181 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.784159 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dhp\" (UniqueName: \"kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp\") pod \"payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:11.828764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.828739 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxtv\" (UniqueName: \"kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv\") pod \"payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.828881 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.828788 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.829522 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.829499 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.885236 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.885213 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxtv\" (UniqueName: \"kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv\") pod \"payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:11.943990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:11.943969 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:12.052202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.052169 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:12.315657 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:40:12.315629 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc77dc8_899b_4759_8aa6_eef13ea98021.slice/crio-f0726a599e3900fdaa4114f568a768ef1c0b3b9ffb5e022d10556ec76d6c385b WatchSource:0}: Error finding container f0726a599e3900fdaa4114f568a768ef1c0b3b9ffb5e022d10556ec76d6c385b: Status 404 returned error can't find the container with id f0726a599e3900fdaa4114f568a768ef1c0b3b9ffb5e022d10556ec76d6c385b Nov 01 06:40:12.316019 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.315999 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf"] Nov 01 06:40:12.428187 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:40:12.428160 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc046a106_931a_4cbc_b490_5ce3edc652bb.slice/crio-7c81f69c622dc9cb974e51dd01e3c705de9d149938db680687083ccd421898e1 WatchSource:0}: Error finding container 7c81f69c622dc9cb974e51dd01e3c705de9d149938db680687083ccd421898e1: Status 404 returned error can't find the container with id 7c81f69c622dc9cb974e51dd01e3c705de9d149938db680687083ccd421898e1 Nov 01 06:40:12.428564 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.428547 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh"] Nov 01 06:40:12.636552 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.636521 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" event={"ID":"6fc77dc8-899b-4759-8aa6-eef13ea98021","Type":"ContainerStarted","Data":"506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4"} Nov 01 06:40:12.636928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.636579 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:12.636928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.636603 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" event={"ID":"6fc77dc8-899b-4759-8aa6-eef13ea98021","Type":"ContainerStarted","Data":"f0726a599e3900fdaa4114f568a768ef1c0b3b9ffb5e022d10556ec76d6c385b"} Nov 01 06:40:12.638234 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.638214 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" event={"ID":"c046a106-931a-4cbc-b490-5ce3edc652bb","Type":"ContainerStarted","Data":"f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b"} Nov 01 06:40:12.638353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.638251 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" event={"ID":"c046a106-931a-4cbc-b490-5ce3edc652bb","Type":"ContainerStarted","Data":"7c81f69c622dc9cb974e51dd01e3c705de9d149938db680687083ccd421898e1"} Nov 01 06:40:12.638353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.638295 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:12.752316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.752265 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" podStartSLOduration=1.752254197 podStartE2EDuration="1.752254197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:12.74916029 +0000 UTC m=+1828.301949483" watchObservedRunningTime="2026-11-01 06:40:12.752254197 +0000 UTC m=+1828.305043388" Nov 01 06:40:12.881303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:12.877843 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" podStartSLOduration=1.877823349 podStartE2EDuration="1.877823349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:40:12.875832949 +0000 UTC m=+1828.428622143" watchObservedRunningTime="2026-11-01 06:40:12.877823349 +0000 UTC m=+1828.430612542" Nov 01 06:40:16.774768 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:16.774734 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:16.786726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:16.786703 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:17.677551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:17.677524 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:40:23.242400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:23.242361 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj"] Nov 01 06:40:23.242844 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:23.242696 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" containerID="cri-o://918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78" gracePeriod=30 Nov 01 06:40:23.647926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:23.647875 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:40:23.648235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:23.648218 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:40:28.523373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.523321 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:40:28.642196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642174 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642211 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642246 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642334 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642395 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642440 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642468 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642511 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642539 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs\") pod \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\" (UID: \"304cf2e2-d0ab-4b7e-aabc-0a105ce15233\") " Nov 01 06:40:28.642721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642702 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.642817 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642794 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.642875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642799 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:28.643083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.642896 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.643191 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.643162 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.643293 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.643203 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.643293 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.643217 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.643293 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.643230 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.643530 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.643504 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.644623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.644598 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:28.644723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.644602 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:28.644723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.644639 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:28.644831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.644759 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms" pod "304cf2e2-d0ab-4b7e-aabc-0a105ce15233" (UID: "304cf2e2-d0ab-4b7e-aabc-0a105ce15233"). InnerVolumeSpecName "kube-api-access-8hdms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:28.732305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.732243 4914 generic.go:356] "Generic (PLEG): container finished" podID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerID="918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78" exitCode=0 Nov 01 06:40:28.732438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.732320 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" Nov 01 06:40:28.732438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.732310 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" event={"ID":"304cf2e2-d0ab-4b7e-aabc-0a105ce15233","Type":"ContainerDied","Data":"918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78"} Nov 01 06:40:28.732438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.732420 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj" event={"ID":"304cf2e2-d0ab-4b7e-aabc-0a105ce15233","Type":"ContainerDied","Data":"fb7567301b1625d7c0625a4f353da5239c54da79011f3aed223b5cf712182ad2"} Nov 01 06:40:28.732622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.732452 4914 scope.go:122] "RemoveContainer" containerID="918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78" Nov 01 06:40:28.745839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.745817 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.745908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.745848 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdms\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-kube-api-access-8hdms\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.745908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.745864 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.745908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.745879 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.745908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.745893 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304cf2e2-d0ab-4b7e-aabc-0a105ce15233-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:28.747419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.747396 4914 scope.go:122] "RemoveContainer" containerID="918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78" Nov 01 06:40:28.747740 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:40:28.747714 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78\": container with ID starting with 918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78 not found: ID does not exist" containerID="918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78" Nov 01 06:40:28.747888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.747852 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78"} err="failed to get container status \"918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78\": rpc error: code = NotFound desc = could not find container \"918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78\": container with ID starting with 918e5015d53ddc95ed2506d3ec7d8fc903931acd8da67b004a2ec28bb0d29f78 not found: ID does not exist" Nov 01 06:40:28.899795 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.899768 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj"] Nov 01 06:40:28.903232 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:28.903206 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-476fj"] Nov 01 06:40:30.616848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:30.616809 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" path="/var/lib/kubelet/pods/304cf2e2-d0ab-4b7e-aabc-0a105ce15233/volumes" Nov 01 06:40:31.518810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:31.518781 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l"] Nov 01 06:40:31.519027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:31.518987 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" containerID="cri-o://7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad" gracePeriod=30 Nov 01 06:40:36.778670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.778648 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:40:36.782074 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.782052 4914 generic.go:356] "Generic (PLEG): container finished" podID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerID="7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad" exitCode=0 Nov 01 06:40:36.782192 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.782091 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" Nov 01 06:40:36.782192 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.782133 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" event={"ID":"19b4bc82-eb35-4d86-bba8-48e662e71f09","Type":"ContainerDied","Data":"7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad"} Nov 01 06:40:36.782192 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.782184 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" event={"ID":"19b4bc82-eb35-4d86-bba8-48e662e71f09","Type":"ContainerDied","Data":"87e3e16a648f6628a9fead08b51762fa89e1df48812302016eb5b8c45af0b126"} Nov 01 06:40:36.782333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.782210 4914 scope.go:122] "RemoveContainer" containerID="7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad" Nov 01 06:40:36.791886 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.791857 4914 scope.go:122] "RemoveContainer" containerID="7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad" Nov 01 06:40:36.792141 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:40:36.792124 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad\": container with ID starting with 7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad not found: ID does not exist" containerID="7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad" Nov 01 06:40:36.792206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.792150 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad"} err="failed to get container status \"7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad\": rpc error: code = NotFound desc = could not find container \"7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad\": container with ID starting with 7a0c36eaab2614450ccbb8fc0f9c80c689873f6fb9d35d239e5c8c438d39b8ad not found: ID does not exist" Nov 01 06:40:36.926472 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926443 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.926605 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926561 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.926605 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926592 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.926684 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926657 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.926733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926707 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.926784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926714 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.926841 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926828 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.927004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.926981 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.927122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.927093 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.927313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.927204 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.927461 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.927442 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.927548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.927476 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.927548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.927484 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert\") pod \"19b4bc82-eb35-4d86-bba8-48e662e71f09\" (UID: \"19b4bc82-eb35-4d86-bba8-48e662e71f09\") " Nov 01 06:40:36.927948 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.927758 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:40:36.928119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.928096 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.928196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.928123 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.928196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.928140 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.928196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.928152 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4bc82-eb35-4d86-bba8-48e662e71f09-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.928196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.928160 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:36.929256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.929229 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:40:36.929411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.929385 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:40:36.929683 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.929662 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:36.929947 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:36.929918 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9" pod "19b4bc82-eb35-4d86-bba8-48e662e71f09" (UID: "19b4bc82-eb35-4d86-bba8-48e662e71f09"). InnerVolumeSpecName "kube-api-access-7v4x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:40:37.029803 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.029770 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4x9\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-kube-api-access-7v4x9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.029931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.029808 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.029931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.029823 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.029931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.029837 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4bc82-eb35-4d86-bba8-48e662e71f09-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:40:37.203090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.203047 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l"] Nov 01 06:40:37.206708 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.206680 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l"] Nov 01 06:40:37.592295 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:40:37.592247 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:40:37.697141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.697111 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 06:40:37.697302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:37.697168 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2bx8l" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 06:40:38.616551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:40:38.616519 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" path="/var/lib/kubelet/pods/19b4bc82-eb35-4d86-bba8-48e662e71f09/volumes" Nov 01 06:41:07.597562 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:41:07.597526 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:41:13.051822 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:13.051788 4914 generic.go:356] "Generic (PLEG): container finished" podID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerID="d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172" exitCode=0 Nov 01 06:41:13.052211 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:13.051860 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" event={"ID":"a714f6bb-9938-46d3-8ee1-dab54ec89a84","Type":"ContainerDied","Data":"d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172"} Nov 01 06:41:13.052211 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:13.052165 4914 scope.go:122] "RemoveContainer" containerID="d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172" Nov 01 06:41:13.053494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:13.053465 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:41:13.053556 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:13.053531 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:41:13.053592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:13.053568 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:41:14.076755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:14.076722 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" event={"ID":"a714f6bb-9938-46d3-8ee1-dab54ec89a84","Type":"ContainerStarted","Data":"46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9"} Nov 01 06:41:14.823021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:14.822988 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:41:14.824167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:14.824145 4914 patch_prober.go:29] interesting pod/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 06:41:14.824222 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:14.824203 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 06:41:15.822719 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:15.822689 4914 patch_prober.go:29] interesting pod/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 06:41:15.823079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:15.822740 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 06:41:16.822824 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:16.822789 4914 patch_prober.go:29] interesting pod/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 06:41:16.823169 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:16.822847 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 06:41:17.823586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:17.823552 4914 patch_prober.go:29] interesting pod/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 01 06:41:17.823933 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:17.823626 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 01 06:41:18.839153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:18.839117 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:41:18.839987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:18.839966 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:41:18.840576 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:18.840559 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:41:37.591342 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:41:37.591304 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:41:40.909652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:40.909619 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn"] Nov 01 06:41:43.923176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:43.923141 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc"] Nov 01 06:41:45.910611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:45.910571 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" podUID="8d69385c-7a3b-423f-85d1-36c257039dd8" containerName="payload-processing" containerID="cri-o://f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c" gracePeriod=25 Nov 01 06:41:46.167605 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.167551 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:41:46.283741 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.283607 4914 generic.go:356] "Generic (PLEG): container finished" podID="8d69385c-7a3b-423f-85d1-36c257039dd8" containerID="f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c" exitCode=0 Nov 01 06:41:46.283741 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.283657 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" event={"ID":"8d69385c-7a3b-423f-85d1-36c257039dd8","Type":"ContainerDied","Data":"f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c"} Nov 01 06:41:46.283741 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.283668 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" Nov 01 06:41:46.283980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.283762 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn" event={"ID":"8d69385c-7a3b-423f-85d1-36c257039dd8","Type":"ContainerDied","Data":"a2520a43b24ef1b4dcba53a0e975826a36a2a06f0ac850e063d0a4ee5f6c0d21"} Nov 01 06:41:46.283980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.283792 4914 scope.go:122] "RemoveContainer" containerID="f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c" Nov 01 06:41:46.295223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.295207 4914 scope.go:122] "RemoveContainer" containerID="f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c" Nov 01 06:41:46.295547 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:41:46.295519 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c\": container with ID starting with f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c not found: ID does not exist" containerID="f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c" Nov 01 06:41:46.295640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.295556 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c"} err="failed to get container status \"f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c\": rpc error: code = NotFound desc = could not find container \"f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c\": container with ID starting with f48ad790a918e8e76a148bce936c5e6cc2ee8fdde9981cb1a845670114328a9c not found: ID does not exist" Nov 01 06:41:46.325026 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325005 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn\") pod \"8d69385c-7a3b-423f-85d1-36c257039dd8\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " Nov 01 06:41:46.325144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325057 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume\") pod \"8d69385c-7a3b-423f-85d1-36c257039dd8\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " Nov 01 06:41:46.325206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325156 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume\") pod \"8d69385c-7a3b-423f-85d1-36c257039dd8\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " Nov 01 06:41:46.325835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325257 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials\") pod \"8d69385c-7a3b-423f-85d1-36c257039dd8\" (UID: \"8d69385c-7a3b-423f-85d1-36c257039dd8\") " Nov 01 06:41:46.325835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325513 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume" pod "8d69385c-7a3b-423f-85d1-36c257039dd8" (UID: "8d69385c-7a3b-423f-85d1-36c257039dd8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:41:46.325835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325806 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume" pod "8d69385c-7a3b-423f-85d1-36c257039dd8" (UID: "8d69385c-7a3b-423f-85d1-36c257039dd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:41:46.326063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.325877 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:46.327894 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.327873 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials" pod "8d69385c-7a3b-423f-85d1-36c257039dd8" (UID: "8d69385c-7a3b-423f-85d1-36c257039dd8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:41:46.327977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.327894 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn" pod "8d69385c-7a3b-423f-85d1-36c257039dd8" (UID: "8d69385c-7a3b-423f-85d1-36c257039dd8"). InnerVolumeSpecName "kube-api-access-28wwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:41:46.426742 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.426678 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28wwn\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-kube-api-access-28wwn\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:46.426742 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.426704 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d69385c-7a3b-423f-85d1-36c257039dd8-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:46.426742 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.426716 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d69385c-7a3b-423f-85d1-36c257039dd8-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:46.704368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.704288 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn"] Nov 01 06:41:46.754672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:46.754646 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-6f7c7xzdn"] Nov 01 06:41:48.616149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:48.616115 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d69385c-7a3b-423f-85d1-36c257039dd8" path="/var/lib/kubelet/pods/8d69385c-7a3b-423f-85d1-36c257039dd8/volumes" Nov 01 06:41:48.923753 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:48.923658 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" podUID="1e095a19-cee8-4b1d-81e9-022d98de1cd9" containerName="payload-processing" containerID="cri-o://0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635" gracePeriod=25 Nov 01 06:41:49.202340 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.202317 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:41:49.273888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.273863 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf\") pod \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " Nov 01 06:41:49.274003 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.273989 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume\") pod \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " Nov 01 06:41:49.274095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.274081 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials\") pod \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " Nov 01 06:41:49.274153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.274110 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume\") pod \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\" (UID: \"1e095a19-cee8-4b1d-81e9-022d98de1cd9\") " Nov 01 06:41:49.274451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.274356 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume" pod "1e095a19-cee8-4b1d-81e9-022d98de1cd9" (UID: "1e095a19-cee8-4b1d-81e9-022d98de1cd9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:41:49.274711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.274508 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume" pod "1e095a19-cee8-4b1d-81e9-022d98de1cd9" (UID: "1e095a19-cee8-4b1d-81e9-022d98de1cd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:41:49.274711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.274548 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:49.275963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.275944 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf" pod "1e095a19-cee8-4b1d-81e9-022d98de1cd9" (UID: "1e095a19-cee8-4b1d-81e9-022d98de1cd9"). InnerVolumeSpecName "kube-api-access-9m6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:41:49.276113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.276093 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials" pod "1e095a19-cee8-4b1d-81e9-022d98de1cd9" (UID: "1e095a19-cee8-4b1d-81e9-022d98de1cd9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:41:49.303300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.303253 4914 generic.go:356] "Generic (PLEG): container finished" podID="1e095a19-cee8-4b1d-81e9-022d98de1cd9" containerID="0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635" exitCode=0 Nov 01 06:41:49.303380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.303309 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" Nov 01 06:41:49.303380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.303309 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" event={"ID":"1e095a19-cee8-4b1d-81e9-022d98de1cd9","Type":"ContainerDied","Data":"0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635"} Nov 01 06:41:49.303380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.303362 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc" event={"ID":"1e095a19-cee8-4b1d-81e9-022d98de1cd9","Type":"ContainerDied","Data":"0a2915c8d88a0dfe11bd9c9861d50ca9f1d36c83bf8a395379875b337d8c6175"} Nov 01 06:41:49.303476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.303383 4914 scope.go:122] "RemoveContainer" containerID="0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635" Nov 01 06:41:49.312584 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.312566 4914 scope.go:122] "RemoveContainer" containerID="0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635" Nov 01 06:41:49.312824 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:41:49.312810 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635\": container with ID starting with 0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635 not found: ID does not exist" containerID="0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635" Nov 01 06:41:49.312890 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.312831 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635"} err="failed to get container status \"0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635\": rpc error: code = NotFound desc = could not find container \"0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635\": container with ID starting with 0579f1714774b376b21e80fc7af521af396c2948adb69d5f399abb1b62f1c635 not found: ID does not exist" Nov 01 06:41:49.375225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.375199 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:49.375225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.375221 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e095a19-cee8-4b1d-81e9-022d98de1cd9-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:49.375225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.375230 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6cf\" (UniqueName: \"kubernetes.io/projected/1e095a19-cee8-4b1d-81e9-022d98de1cd9-kube-api-access-9m6cf\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:41:49.428853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.428787 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc"] Nov 01 06:41:49.434299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:49.434252 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdbdd2gc"] Nov 01 06:41:50.616184 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:41:50.616152 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e095a19-cee8-4b1d-81e9-022d98de1cd9" path="/var/lib/kubelet/pods/1e095a19-cee8-4b1d-81e9-022d98de1cd9/volumes" Nov 01 06:42:07.591741 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:42:07.591664 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:42:37.592807 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:42:37.592764 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:42:44.666195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:44.666147 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf"] Nov 01 06:42:44.769471 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:44.769438 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh"] Nov 01 06:42:47.334426 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:47.334397 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl"] Nov 01 06:42:47.541613 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:47.541578 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2"] Nov 01 06:42:47.541839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:47.541815 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" containerID="cri-o://ab477cf78ef3da68601c4fe9564a4bc8ee3f38216bc9850ba3418ed010743269" gracePeriod=30 Nov 01 06:42:49.667291 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:49.667223 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" podUID="6fc77dc8-899b-4759-8aa6-eef13ea98021" containerName="payload-processing" containerID="cri-o://506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4" gracePeriod=25 Nov 01 06:42:49.769845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:49.769781 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" podUID="c046a106-931a-4cbc-b490-5ce3edc652bb" containerName="payload-pre-processing" containerID="cri-o://f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b" gracePeriod=25 Nov 01 06:42:50.125820 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.125791 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:42:50.208428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.207435 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume\") pod \"6fc77dc8-899b-4759-8aa6-eef13ea98021\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " Nov 01 06:42:50.208428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.207614 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv\" (UniqueName: \"kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv\") pod \"6fc77dc8-899b-4759-8aa6-eef13ea98021\" (UID: \"6fc77dc8-899b-4759-8aa6-eef13ea98021\") " Nov 01 06:42:50.209534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.209045 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume" pod "6fc77dc8-899b-4759-8aa6-eef13ea98021" (UID: "6fc77dc8-899b-4759-8aa6-eef13ea98021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:50.215149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.215095 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv" pod "6fc77dc8-899b-4759-8aa6-eef13ea98021" (UID: "6fc77dc8-899b-4759-8aa6-eef13ea98021"). InnerVolumeSpecName "kube-api-access-gwxtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:50.233439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.233418 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:42:50.309303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.309254 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp\" (UniqueName: \"kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp\") pod \"c046a106-931a-4cbc-b490-5ce3edc652bb\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " Nov 01 06:42:50.309491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.309397 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume\") pod \"c046a106-931a-4cbc-b490-5ce3edc652bb\" (UID: \"c046a106-931a-4cbc-b490-5ce3edc652bb\") " Nov 01 06:42:50.309797 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.309773 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxtv\" (UniqueName: \"kubernetes.io/projected/6fc77dc8-899b-4759-8aa6-eef13ea98021-kube-api-access-gwxtv\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:50.309884 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.309801 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc77dc8-899b-4759-8aa6-eef13ea98021-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:50.310148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.310117 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume" pod "c046a106-931a-4cbc-b490-5ce3edc652bb" (UID: "c046a106-931a-4cbc-b490-5ce3edc652bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:50.312037 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.312019 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp" pod "c046a106-931a-4cbc-b490-5ce3edc652bb" (UID: "c046a106-931a-4cbc-b490-5ce3edc652bb"). InnerVolumeSpecName "kube-api-access-27dhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:50.411437 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.411125 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c046a106-931a-4cbc-b490-5ce3edc652bb-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:50.411437 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.411167 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27dhp\" (UniqueName: \"kubernetes.io/projected/c046a106-931a-4cbc-b490-5ce3edc652bb-kube-api-access-27dhp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:50.722396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.722362 4914 generic.go:356] "Generic (PLEG): container finished" podID="6fc77dc8-899b-4759-8aa6-eef13ea98021" containerID="506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4" exitCode=0 Nov 01 06:42:50.722802 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.722419 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" Nov 01 06:42:50.722802 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.722452 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" event={"ID":"6fc77dc8-899b-4759-8aa6-eef13ea98021","Type":"ContainerDied","Data":"506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4"} Nov 01 06:42:50.722802 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.722546 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf" event={"ID":"6fc77dc8-899b-4759-8aa6-eef13ea98021","Type":"ContainerDied","Data":"f0726a599e3900fdaa4114f568a768ef1c0b3b9ffb5e022d10556ec76d6c385b"} Nov 01 06:42:50.722802 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.722580 4914 scope.go:122] "RemoveContainer" containerID="506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4" Nov 01 06:42:50.724098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.724058 4914 generic.go:356] "Generic (PLEG): container finished" podID="c046a106-931a-4cbc-b490-5ce3edc652bb" containerID="f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b" exitCode=0 Nov 01 06:42:50.724163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.724119 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" Nov 01 06:42:50.724207 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.724172 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" event={"ID":"c046a106-931a-4cbc-b490-5ce3edc652bb","Type":"ContainerDied","Data":"f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b"} Nov 01 06:42:50.724242 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.724207 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh" event={"ID":"c046a106-931a-4cbc-b490-5ce3edc652bb","Type":"ContainerDied","Data":"7c81f69c622dc9cb974e51dd01e3c705de9d149938db680687083ccd421898e1"} Nov 01 06:42:50.733073 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.733052 4914 scope.go:122] "RemoveContainer" containerID="506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4" Nov 01 06:42:50.733417 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:42:50.733396 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4\": container with ID starting with 506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4 not found: ID does not exist" containerID="506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4" Nov 01 06:42:50.733495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.733423 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4"} err="failed to get container status \"506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4\": rpc error: code = NotFound desc = could not find container \"506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4\": container with ID starting with 506cc053ffc9f46f9b19bbb14298e886f0c65cdc96464edc78c8a85e55903af4 not found: ID does not exist" Nov 01 06:42:50.733495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.733441 4914 scope.go:122] "RemoveContainer" containerID="f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b" Nov 01 06:42:50.742159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.742141 4914 scope.go:122] "RemoveContainer" containerID="f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b" Nov 01 06:42:50.742477 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:42:50.742459 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b\": container with ID starting with f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b not found: ID does not exist" containerID="f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b" Nov 01 06:42:50.742557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.742487 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b"} err="failed to get container status \"f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b\": rpc error: code = NotFound desc = could not find container \"f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b\": container with ID starting with f70dd40330b86ee4e0570b130f4de83a10a6ebe7c459e9256f013a3dadca6f8b not found: ID does not exist" Nov 01 06:42:50.753016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.752960 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:42:50.753153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.753034 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:42:50.891610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.891460 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf"] Nov 01 06:42:50.897293 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:50.897238 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c66acdd-7889576f8f-x6tqf"] Nov 01 06:42:51.008376 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:51.008343 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh"] Nov 01 06:42:51.012565 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:51.012535 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c66acdd-9bd86b959-9gzkh"] Nov 01 06:42:52.335555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.335500 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" podUID="62f8efa0-55e0-441f-90d5-68c347df8759" containerName="payload-processing" containerID="cri-o://f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d" gracePeriod=25 Nov 01 06:42:52.619357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.619325 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc77dc8-899b-4759-8aa6-eef13ea98021" path="/var/lib/kubelet/pods/6fc77dc8-899b-4759-8aa6-eef13ea98021/volumes" Nov 01 06:42:52.619795 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.619780 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c046a106-931a-4cbc-b490-5ce3edc652bb" path="/var/lib/kubelet/pods/c046a106-931a-4cbc-b490-5ce3edc652bb/volumes" Nov 01 06:42:52.673752 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.673724 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:42:52.731455 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.731422 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume\") pod \"62f8efa0-55e0-441f-90d5-68c347df8759\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " Nov 01 06:42:52.731608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.731472 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume\") pod \"62f8efa0-55e0-441f-90d5-68c347df8759\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " Nov 01 06:42:52.731608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.731575 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9\") pod \"62f8efa0-55e0-441f-90d5-68c347df8759\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " Nov 01 06:42:52.731701 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.731652 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials\") pod \"62f8efa0-55e0-441f-90d5-68c347df8759\" (UID: \"62f8efa0-55e0-441f-90d5-68c347df8759\") " Nov 01 06:42:52.731880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.731859 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume" pod "62f8efa0-55e0-441f-90d5-68c347df8759" (UID: "62f8efa0-55e0-441f-90d5-68c347df8759"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:52.731970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.731943 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume" pod "62f8efa0-55e0-441f-90d5-68c347df8759" (UID: "62f8efa0-55e0-441f-90d5-68c347df8759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:52.732147 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.732120 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.732205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.732145 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f8efa0-55e0-441f-90d5-68c347df8759-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.735215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.735191 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials" pod "62f8efa0-55e0-441f-90d5-68c347df8759" (UID: "62f8efa0-55e0-441f-90d5-68c347df8759"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:52.735357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.735221 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9" pod "62f8efa0-55e0-441f-90d5-68c347df8759" (UID: "62f8efa0-55e0-441f-90d5-68c347df8759"). InnerVolumeSpecName "kube-api-access-hq8z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:52.741153 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.741125 4914 generic.go:356] "Generic (PLEG): container finished" podID="62f8efa0-55e0-441f-90d5-68c347df8759" containerID="f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d" exitCode=0 Nov 01 06:42:52.741301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.741250 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" Nov 01 06:42:52.741380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.741351 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" event={"ID":"62f8efa0-55e0-441f-90d5-68c347df8759","Type":"ContainerDied","Data":"f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d"} Nov 01 06:42:52.741431 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.741420 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl" event={"ID":"62f8efa0-55e0-441f-90d5-68c347df8759","Type":"ContainerDied","Data":"579a16238f49b8c6f215fea011a2d2a2134d0f07de47959a02cf807ed41b9e91"} Nov 01 06:42:52.741478 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.741449 4914 scope.go:122] "RemoveContainer" containerID="f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d" Nov 01 06:42:52.743905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.743875 4914 generic.go:356] "Generic (PLEG): container finished" podID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerID="ab477cf78ef3da68601c4fe9564a4bc8ee3f38216bc9850ba3418ed010743269" exitCode=0 Nov 01 06:42:52.744057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.744039 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" event={"ID":"5d707cb0-5872-4d7d-893b-49b553872b0c","Type":"ContainerDied","Data":"ab477cf78ef3da68601c4fe9564a4bc8ee3f38216bc9850ba3418ed010743269"} Nov 01 06:42:52.753818 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.753794 4914 scope.go:122] "RemoveContainer" containerID="f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d" Nov 01 06:42:52.754261 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:42:52.754151 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d\": container with ID starting with f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d not found: ID does not exist" containerID="f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d" Nov 01 06:42:52.754261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.754193 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d"} err="failed to get container status \"f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d\": rpc error: code = NotFound desc = could not find container \"f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d\": container with ID starting with f305afd976abbaca53971b2f002b7572138223fddc305e3df749774ef3ac236d not found: ID does not exist" Nov 01 06:42:52.796482 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.796454 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:42:52.832652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832617 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.832836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832707 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.832836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832768 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.832836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832797 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.832836 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832827 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.833112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832896 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.833112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.832959 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.833112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833066 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:42:52.833254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833101 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.833254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833163 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:42:52.833254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833238 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:42:52.833439 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833259 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket\") pod \"5d707cb0-5872-4d7d-893b-49b553872b0c\" (UID: \"5d707cb0-5872-4d7d-893b-49b553872b0c\") " Nov 01 06:42:52.833506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833481 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:42:52.833568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833542 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:42:52.834039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.833956 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.834039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.834022 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.834039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.834040 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8z9\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-kube-api-access-hq8z9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.834294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.834055 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d707cb0-5872-4d7d-893b-49b553872b0c-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.834294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.834070 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.834294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.834083 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.834294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.834097 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f8efa0-55e0-441f-90d5-68c347df8759-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.835158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.835135 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:42:52.835334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.835304 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:42:52.835426 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.835393 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "kube-api-access-s4pg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:52.835495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.835480 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token" pod "5d707cb0-5872-4d7d-893b-49b553872b0c" (UID: "5d707cb0-5872-4d7d-893b-49b553872b0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:42:52.902623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.902567 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl"] Nov 01 06:42:52.905021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.904998 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-6fbfkd7pl"] Nov 01 06:42:52.935095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.935058 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.935095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.935091 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.935095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.935102 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4pg9\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-kube-api-access-s4pg9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:52.935334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:52.935115 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d707cb0-5872-4d7d-893b-49b553872b0c-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:42:53.752342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:53.752249 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" Nov 01 06:42:53.752342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:53.752295 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2" event={"ID":"5d707cb0-5872-4d7d-893b-49b553872b0c","Type":"ContainerDied","Data":"7aba113949314ad26d695b7de560ab0e95ddbbc79d1629a0a3c0abcc168be3d8"} Nov 01 06:42:53.752839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:53.752352 4914 scope.go:122] "RemoveContainer" containerID="ab477cf78ef3da68601c4fe9564a4bc8ee3f38216bc9850ba3418ed010743269" Nov 01 06:42:53.968380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:53.968343 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2"] Nov 01 06:42:53.971606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:53.971580 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-2pjf2"] Nov 01 06:42:54.398562 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.398525 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq"] Nov 01 06:42:54.399106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399091 4914 cpu_manager.go:414] "containerMap: removing container" podUID="6fc77dc8-899b-4759-8aa6-eef13ea98021" containerName="payload-processing" Nov 01 06:42:54.399154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399108 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc77dc8-899b-4759-8aa6-eef13ea98021" containerName="payload-processing" Nov 01 06:42:54.399154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399134 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" Nov 01 06:42:54.399154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399139 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" Nov 01 06:42:54.399154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399145 4914 cpu_manager.go:414] "containerMap: removing container" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" Nov 01 06:42:54.399154 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399151 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399160 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1e095a19-cee8-4b1d-81e9-022d98de1cd9" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399165 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e095a19-cee8-4b1d-81e9-022d98de1cd9" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399173 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c046a106-931a-4cbc-b490-5ce3edc652bb" containerName="payload-pre-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399177 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c046a106-931a-4cbc-b490-5ce3edc652bb" containerName="payload-pre-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399184 4914 cpu_manager.go:414] "containerMap: removing container" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399189 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399195 4914 cpu_manager.go:414] "containerMap: removing container" podUID="8d69385c-7a3b-423f-85d1-36c257039dd8" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399199 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d69385c-7a3b-423f-85d1-36c257039dd8" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399211 4914 cpu_manager.go:414] "containerMap: removing container" podUID="62f8efa0-55e0-441f-90d5-68c347df8759" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399216 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f8efa0-55e0-441f-90d5-68c347df8759" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399318 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" containerName="istio-proxy" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399328 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b4bc82-eb35-4d86-bba8-48e662e71f09" containerName="istio-proxy" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399335 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e095a19-cee8-4b1d-81e9-022d98de1cd9" containerName="payload-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399341 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c046a106-931a-4cbc-b490-5ce3edc652bb" containerName="payload-pre-processing" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399347 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="304cf2e2-d0ab-4b7e-aabc-0a105ce15233" containerName="istio-proxy" Nov 01 06:42:54.399348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399353 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d69385c-7a3b-423f-85d1-36c257039dd8" containerName="payload-processing" Nov 01 06:42:54.399822 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399361 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f8efa0-55e0-441f-90d5-68c347df8759" containerName="payload-processing" Nov 01 06:42:54.399822 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.399369 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc77dc8-899b-4759-8aa6-eef13ea98021" containerName="payload-processing" Nov 01 06:42:54.403935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.403919 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.451770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.451735 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.451770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.451775 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.451976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.451792 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.451976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.451813 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.451976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.451936 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59wm\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.452073 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.451981 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.452073 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.452007 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.452137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.452079 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.452172 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.452136 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.461982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.461954 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq"] Nov 01 06:42:54.553640 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553601 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.553808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553647 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.553808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553667 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.553808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553779 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553826 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59wm\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553943 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.553979 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554049 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554255 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554083 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554255 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554103 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554323 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554398 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554414 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.554511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.554429 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.556767 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.556751 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.557102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.557081 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.610954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.610915 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.611147 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.611124 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59wm\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm\") pod \"maas-default-gateway-openshift-default-7d9897b54c-d9bdq\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:54.625360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.625324 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d707cb0-5872-4d7d-893b-49b553872b0c" path="/var/lib/kubelet/pods/5d707cb0-5872-4d7d-893b-49b553872b0c/volumes" Nov 01 06:42:54.625819 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.625804 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f8efa0-55e0-441f-90d5-68c347df8759" path="/var/lib/kubelet/pods/62f8efa0-55e0-441f-90d5-68c347df8759/volumes" Nov 01 06:42:54.717004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:54.716908 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:55.164315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.164287 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq"] Nov 01 06:42:55.165238 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:42:55.165203 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f58a99_b7a8_4c80_bfe1_e65c5f593514.slice/crio-bce6f5badc4d2e50025823b4e011c0557be0ef3e96cb5bad7c2d763e6a2fd889 WatchSource:0}: Error finding container bce6f5badc4d2e50025823b4e011c0557be0ef3e96cb5bad7c2d763e6a2fd889: Status 404 returned error can't find the container with id bce6f5badc4d2e50025823b4e011c0557be0ef3e96cb5bad7c2d763e6a2fd889 Nov 01 06:42:55.167577 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.167542 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:42:55.167672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.167624 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:42:55.167756 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.167689 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:42:55.780826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.780785 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" event={"ID":"48f58a99-b7a8-4c80-bfe1-e65c5f593514","Type":"ContainerStarted","Data":"a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333"} Nov 01 06:42:55.781036 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.780852 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" event={"ID":"48f58a99-b7a8-4c80-bfe1-e65c5f593514","Type":"ContainerStarted","Data":"bce6f5badc4d2e50025823b4e011c0557be0ef3e96cb5bad7c2d763e6a2fd889"} Nov 01 06:42:55.899805 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:55.899752 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podStartSLOduration=1.899739917 podStartE2EDuration="1.899739917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:42:55.89779882 +0000 UTC m=+1991.450588024" watchObservedRunningTime="2026-11-01 06:42:55.899739917 +0000 UTC m=+1991.452529109" Nov 01 06:42:56.718080 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:56.718040 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:42:56.719453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:56.719429 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:42:56.719569 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:56.719473 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 06:42:57.718117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:57.718081 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:42:57.718532 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:57.718131 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 06:42:58.718181 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:58.718143 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:42:58.718647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:58.718199 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 06:42:59.717710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:59.717666 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:42:59.717880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:42:59.717725 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 06:43:00.718030 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:00.717991 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:43:00.718412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:00.718042 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 06:43:01.718215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:01.718182 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 01 06:43:01.718592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:01.718239 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 01 06:43:03.212484 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:03.212450 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:43:03.212926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:03.212841 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:43:03.213729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:03.213708 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 06:43:07.591688 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:43:07.591655 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:43:29.932424 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:29.932341 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs"] Nov 01 06:43:29.932914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:29.932543 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" containerID="cri-o://46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9" gracePeriod=30 Nov 01 06:43:33.841856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:33.841817 4914 patch_prober.go:29] interesting pod/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:43:33.842212 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:33.841882 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:43:35.198231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.198179 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:43:35.236395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236372 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236498 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236483 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236539 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236520 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236579 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236566 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236654 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236630 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236691 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236734 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236760 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.236959 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236851 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:35.236959 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236866 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:35.236959 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236871 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:35.236959 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.236902 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert\") pod \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\" (UID: \"a714f6bb-9938-46d3-8ee1-dab54ec89a84\") " Nov 01 06:43:35.237357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237252 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:35.237357 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237270 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:43:35.237470 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237411 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.237470 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237430 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.237580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237446 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.237580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237542 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.237689 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.237626 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.238781 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.238756 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "kube-api-access-5nbkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:35.238971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.238948 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:43:35.239290 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.239259 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:43:35.239356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.239317 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token" pod "a714f6bb-9938-46d3-8ee1-dab54ec89a84" (UID: "a714f6bb-9938-46d3-8ee1-dab54ec89a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:43:35.338262 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.338234 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nbkj\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-kube-api-access-5nbkj\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.338262 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.338260 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.338415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.338303 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:35.338415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:35.338320 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a714f6bb-9938-46d3-8ee1-dab54ec89a84-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:43:36.012007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.011970 4914 generic.go:356] "Generic (PLEG): container finished" podID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerID="46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9" exitCode=0 Nov 01 06:43:36.012167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.012030 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" Nov 01 06:43:36.012167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.012048 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" event={"ID":"a714f6bb-9938-46d3-8ee1-dab54ec89a84","Type":"ContainerDied","Data":"46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9"} Nov 01 06:43:36.012167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.012125 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs" event={"ID":"a714f6bb-9938-46d3-8ee1-dab54ec89a84","Type":"ContainerDied","Data":"9ea9693a9504f333a4fbba06f4da3b1dda48e7cabde74747fe7107c3af8e2fd0"} Nov 01 06:43:36.012167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.012148 4914 scope.go:122] "RemoveContainer" containerID="46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9" Nov 01 06:43:36.022809 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.022792 4914 scope.go:122] "RemoveContainer" containerID="d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172" Nov 01 06:43:36.031106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.031090 4914 scope.go:122] "RemoveContainer" containerID="46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9" Nov 01 06:43:36.031359 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:43:36.031336 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9\": container with ID starting with 46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9 not found: ID does not exist" containerID="46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9" Nov 01 06:43:36.031438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.031369 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9"} err="failed to get container status \"46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9\": rpc error: code = NotFound desc = could not find container \"46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9\": container with ID starting with 46e31abd024701676ed89cfb6c50f6caf1c62e45e316c4501051b83c08a5c9c9 not found: ID does not exist" Nov 01 06:43:36.031438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.031393 4914 scope.go:122] "RemoveContainer" containerID="d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172" Nov 01 06:43:36.031652 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:43:36.031637 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172\": container with ID starting with d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172 not found: ID does not exist" containerID="d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172" Nov 01 06:43:36.031690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.031657 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172"} err="failed to get container status \"d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172\": rpc error: code = NotFound desc = could not find container \"d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172\": container with ID starting with d3719e02f1b59e5ca03b965d38dc36a79ce6b2d8f65d7aed482a014f10ff9172 not found: ID does not exist" Nov 01 06:43:36.177030 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.177004 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs"] Nov 01 06:43:36.226173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.226146 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4c66acdd-openshift-default-974d75d7-p2dfs"] Nov 01 06:43:36.617162 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:36.617131 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" path="/var/lib/kubelet/pods/a714f6bb-9938-46d3-8ee1-dab54ec89a84/volumes" Nov 01 06:43:37.591432 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:43:37.591403 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:43:38.512327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.508759 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr"] Nov 01 06:43:38.512327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.509695 4914 cpu_manager.go:414] "containerMap: removing container" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" Nov 01 06:43:38.512327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.509717 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" Nov 01 06:43:38.512327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.509742 4914 cpu_manager.go:414] "containerMap: removing container" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" Nov 01 06:43:38.512327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.509749 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" Nov 01 06:43:38.512327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.509924 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" Nov 01 06:43:38.515090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.515032 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.550222 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.550190 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr"] Nov 01 06:43:38.596798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.596765 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-030936cd\"" Nov 01 06:43:38.596798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.596766 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-030936cd-dockercfg-b5678\"" Nov 01 06:43:38.600493 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.600472 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume\") pod \"payload-processing-e2e-derive-030936cd-5878568bf9-zblpr\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.600589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.600534 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wx5\" (UniqueName: \"kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5\") pod \"payload-processing-e2e-derive-030936cd-5878568bf9-zblpr\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.660928 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.660902 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2"] Nov 01 06:43:38.661555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.661540 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a714f6bb-9938-46d3-8ee1-dab54ec89a84" containerName="istio-proxy" Nov 01 06:43:38.664791 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.664776 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.701098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.701069 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume\") pod \"payload-processing-e2e-derive-030936cd-5878568bf9-zblpr\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.701204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.701164 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wx5\" (UniqueName: \"kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5\") pod \"payload-processing-e2e-derive-030936cd-5878568bf9-zblpr\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.701721 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.701702 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume\") pod \"payload-processing-e2e-derive-030936cd-5878568bf9-zblpr\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.721546 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.721517 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2"] Nov 01 06:43:38.757518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.757491 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wx5\" (UniqueName: \"kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5\") pod \"payload-processing-e2e-derive-030936cd-5878568bf9-zblpr\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.802321 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.802243 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.802321 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.802293 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kw6q\" (UniqueName: \"kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q\") pod \"payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.854168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.854146 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:38.905646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.905052 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.905646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.905095 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kw6q\" (UniqueName: \"kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q\") pod \"payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.906792 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.906767 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.961744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.961713 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kw6q\" (UniqueName: \"kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q\") pod \"payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:38.979144 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:38.979117 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:39.223680 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:39.223657 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr"] Nov 01 06:43:39.225159 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:43:39.225129 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5defd105_0728_4d9b_8fa9_9e3e9b01f1bc.slice/crio-d004f0cd12c2d188436d0337c5c8f5b02f12c2fef2b0429019898a12a8b9d7ca WatchSource:0}: Error finding container d004f0cd12c2d188436d0337c5c8f5b02f12c2fef2b0429019898a12a8b9d7ca: Status 404 returned error can't find the container with id d004f0cd12c2d188436d0337c5c8f5b02f12c2fef2b0429019898a12a8b9d7ca Nov 01 06:43:39.349454 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:39.349431 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2"] Nov 01 06:43:39.350907 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:43:39.350877 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd56e7d_7b78_4240_b454_b20d66bed003.slice/crio-6d340dada4eacbc03f39705be44c5b67551a6bd38350f8d9d11975787ab82fbb WatchSource:0}: Error finding container 6d340dada4eacbc03f39705be44c5b67551a6bd38350f8d9d11975787ab82fbb: Status 404 returned error can't find the container with id 6d340dada4eacbc03f39705be44c5b67551a6bd38350f8d9d11975787ab82fbb Nov 01 06:43:40.051082 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.051048 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" event={"ID":"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc","Type":"ContainerStarted","Data":"bd5848c4546161b808a57f9b9f9cc7a99b467c7397dc27d2c73bceab7aa744c8"} Nov 01 06:43:40.051525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.051144 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:43:40.051525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.051173 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" event={"ID":"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc","Type":"ContainerStarted","Data":"d004f0cd12c2d188436d0337c5c8f5b02f12c2fef2b0429019898a12a8b9d7ca"} Nov 01 06:43:40.052907 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.052879 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" event={"ID":"2bd56e7d-7b78-4240-b454-b20d66bed003","Type":"ContainerStarted","Data":"5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488"} Nov 01 06:43:40.053016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.052929 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" event={"ID":"2bd56e7d-7b78-4240-b454-b20d66bed003","Type":"ContainerStarted","Data":"6d340dada4eacbc03f39705be44c5b67551a6bd38350f8d9d11975787ab82fbb"} Nov 01 06:43:40.053016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.052961 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:40.179152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.179109 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" podStartSLOduration=2.179096863 podStartE2EDuration="2.179096863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:40.176231046 +0000 UTC m=+2035.729020238" watchObservedRunningTime="2026-11-01 06:43:40.179096863 +0000 UTC m=+2035.731886054" Nov 01 06:43:40.312876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:40.312740 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" podStartSLOduration=2.312719953 podStartE2EDuration="2.312719953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:43:40.3109431 +0000 UTC m=+2035.863732293" watchObservedRunningTime="2026-11-01 06:43:40.312719953 +0000 UTC m=+2035.865509149" Nov 01 06:43:51.061650 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:51.061620 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:43:51.062023 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:43:51.061754 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:44:07.591242 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:44:07.591202 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:44:36.081231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:36.081200 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv"] Nov 01 06:44:37.591368 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:44:37.591333 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:44:44.855453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:44.855421 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:44:44.861131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:44.861114 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:44:51.101382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.101341 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" podUID="aff97354-8e10-41db-b61e-b2828b6dc0c0" containerName="main" containerID="cri-o://048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2" gracePeriod=45 Nov 01 06:44:51.361533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.361476 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:44:51.434653 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434629 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir\") pod \"aff97354-8e10-41db-b61e-b2828b6dc0c0\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " Nov 01 06:44:51.434779 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434662 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache\") pod \"aff97354-8e10-41db-b61e-b2828b6dc0c0\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " Nov 01 06:44:51.434779 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434697 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home\") pod \"aff97354-8e10-41db-b61e-b2828b6dc0c0\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " Nov 01 06:44:51.434898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434798 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm\") pod \"aff97354-8e10-41db-b61e-b2828b6dc0c0\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " Nov 01 06:44:51.434898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434831 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs\") pod \"aff97354-8e10-41db-b61e-b2828b6dc0c0\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " Nov 01 06:44:51.434898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434861 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr\" (UniqueName: \"kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr\") pod \"aff97354-8e10-41db-b61e-b2828b6dc0c0\" (UID: \"aff97354-8e10-41db-b61e-b2828b6dc0c0\") " Nov 01 06:44:51.434898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434872 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir" pod "aff97354-8e10-41db-b61e-b2828b6dc0c0" (UID: "aff97354-8e10-41db-b61e-b2828b6dc0c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:44:51.435060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.434916 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache" pod "aff97354-8e10-41db-b61e-b2828b6dc0c0" (UID: "aff97354-8e10-41db-b61e-b2828b6dc0c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:44:51.435060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.435004 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home" pod "aff97354-8e10-41db-b61e-b2828b6dc0c0" (UID: "aff97354-8e10-41db-b61e-b2828b6dc0c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:44:51.435244 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.435225 4914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-tmp-dir\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:51.435371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.435247 4914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-model-cache\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:51.435371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.435259 4914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-home\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:51.437079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.437054 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm" pod "aff97354-8e10-41db-b61e-b2828b6dc0c0" (UID: "aff97354-8e10-41db-b61e-b2828b6dc0c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:44:51.437079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.437063 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs" pod "aff97354-8e10-41db-b61e-b2828b6dc0c0" (UID: "aff97354-8e10-41db-b61e-b2828b6dc0c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:44:51.437306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.437127 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr" pod "aff97354-8e10-41db-b61e-b2828b6dc0c0" (UID: "aff97354-8e10-41db-b61e-b2828b6dc0c0"). InnerVolumeSpecName "kube-api-access-qngpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:44:51.459246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.459223 4914 generic.go:356] "Generic (PLEG): container finished" podID="aff97354-8e10-41db-b61e-b2828b6dc0c0" containerID="048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2" exitCode=0 Nov 01 06:44:51.459333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.459264 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" Nov 01 06:44:51.459333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.459307 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" event={"ID":"aff97354-8e10-41db-b61e-b2828b6dc0c0","Type":"ContainerDied","Data":"048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2"} Nov 01 06:44:51.459400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.459350 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv" event={"ID":"aff97354-8e10-41db-b61e-b2828b6dc0c0","Type":"ContainerDied","Data":"e3092a4f93ee20329bd5a8f01e1f3d86a29b340de875b96c3b2fe464cd2b70ae"} Nov 01 06:44:51.459400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.459374 4914 scope.go:122] "RemoveContainer" containerID="048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2" Nov 01 06:44:51.468964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.468948 4914 scope.go:122] "RemoveContainer" containerID="048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2" Nov 01 06:44:51.469248 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:44:51.469231 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2\": container with ID starting with 048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2 not found: ID does not exist" containerID="048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2" Nov 01 06:44:51.469446 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.469255 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2"} err="failed to get container status \"048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2\": rpc error: code = NotFound desc = could not find container \"048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2\": container with ID starting with 048b2da2235294989eaa99b3ad310c2ec21c132ef62f4baa0a2bcfd800150fd2 not found: ID does not exist" Nov 01 06:44:51.536205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.536182 4914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aff97354-8e10-41db-b61e-b2828b6dc0c0-dshm\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:51.536205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.536202 4914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aff97354-8e10-41db-b61e-b2828b6dc0c0-tls-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:51.536347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.536211 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngpr\" (UniqueName: \"kubernetes.io/projected/aff97354-8e10-41db-b61e-b2828b6dc0c0-kube-api-access-qngpr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:44:51.584135 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.584105 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv"] Nov 01 06:44:51.587786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:51.587767 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2e0116/auth-test-model-2e0116-kserve-7db55d86d5-ttdtv"] Nov 01 06:44:52.618155 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:44:52.618122 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff97354-8e10-41db-b61e-b2828b6dc0c0" path="/var/lib/kubelet/pods/aff97354-8e10-41db-b61e-b2828b6dc0c0/volumes" Nov 01 06:45:04.644458 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.644383 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv"] Nov 01 06:45:04.645006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.644990 4914 cpu_manager.go:414] "containerMap: removing container" podUID="aff97354-8e10-41db-b61e-b2828b6dc0c0" containerName="main" Nov 01 06:45:04.645053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.645007 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff97354-8e10-41db-b61e-b2828b6dc0c0" containerName="main" Nov 01 06:45:04.645121 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.645112 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff97354-8e10-41db-b61e-b2828b6dc0c0" containerName="main" Nov 01 06:45:04.649625 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.649603 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.679679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.679658 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.679799 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.679718 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cw2w\" (UniqueName: \"kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w\") pod \"payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.754045 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.754015 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8a9760-dockercfg-qwlfq\"" Nov 01 06:45:04.754045 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.754015 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8a9760\"" Nov 01 06:45:04.756573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.756553 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv"] Nov 01 06:45:04.780447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.780421 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.780528 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.780469 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cw2w\" (UniqueName: \"kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w\") pod \"payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.781085 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.781067 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.802334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.802309 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c"] Nov 01 06:45:04.806166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.806152 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:04.861310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.859251 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cw2w\" (UniqueName: \"kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w\") pod \"payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.883297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.881874 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkb99\" (UniqueName: \"kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99\") pod \"payload-processing-e2e-api-a-8a9760-998649d7c-lz57c\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:04.883297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.882047 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8a9760-998649d7c-lz57c\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:04.898348 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.897036 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c"] Nov 01 06:45:04.962990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.962959 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:04.982783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.982761 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8a9760-998649d7c-lz57c\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:04.982942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.982806 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkb99\" (UniqueName: \"kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99\") pod \"payload-processing-e2e-api-a-8a9760-998649d7c-lz57c\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:04.983750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:04.983721 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8a9760-998649d7c-lz57c\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:05.041826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.041795 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkb99\" (UniqueName: \"kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99\") pod \"payload-processing-e2e-api-a-8a9760-998649d7c-lz57c\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:05.119811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.119779 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:05.344475 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.344453 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv"] Nov 01 06:45:05.346212 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:45:05.346181 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103a6ac1_a23b_4cb9_8db9_124d33683828.slice/crio-84d24d9ff284f1bf7052f0f614a3ff4451b05c011aaca6f061fff56932393b69 WatchSource:0}: Error finding container 84d24d9ff284f1bf7052f0f614a3ff4451b05c011aaca6f061fff56932393b69: Status 404 returned error can't find the container with id 84d24d9ff284f1bf7052f0f614a3ff4451b05c011aaca6f061fff56932393b69 Nov 01 06:45:05.495183 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.495113 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c"] Nov 01 06:45:05.499709 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:45:05.499683 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85082994_b04d_4774_bb1e_42119ba6b298.slice/crio-b9333c134eb3220e719ca454d92735d80fdf2a7063f28c0664967ac81bc63ad1 WatchSource:0}: Error finding container b9333c134eb3220e719ca454d92735d80fdf2a7063f28c0664967ac81bc63ad1: Status 404 returned error can't find the container with id b9333c134eb3220e719ca454d92735d80fdf2a7063f28c0664967ac81bc63ad1 Nov 01 06:45:05.569339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.569309 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" event={"ID":"103a6ac1-a23b-4cb9-8db9-124d33683828","Type":"ContainerStarted","Data":"4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1"} Nov 01 06:45:05.569452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.569369 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" event={"ID":"103a6ac1-a23b-4cb9-8db9-124d33683828","Type":"ContainerStarted","Data":"84d24d9ff284f1bf7052f0f614a3ff4451b05c011aaca6f061fff56932393b69"} Nov 01 06:45:05.569452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.569441 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:05.571130 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.571102 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" event={"ID":"85082994-b04d-4774-bb1e-42119ba6b298","Type":"ContainerStarted","Data":"e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338"} Nov 01 06:45:05.571239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.571157 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:05.571239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.571183 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" event={"ID":"85082994-b04d-4774-bb1e-42119ba6b298","Type":"ContainerStarted","Data":"b9333c134eb3220e719ca454d92735d80fdf2a7063f28c0664967ac81bc63ad1"} Nov 01 06:45:05.685468 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.685420 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" podStartSLOduration=1.6854036 podStartE2EDuration="1.6854036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:05.684246735 +0000 UTC m=+2121.237035926" watchObservedRunningTime="2026-11-01 06:45:05.6854036 +0000 UTC m=+2121.238192794" Nov 01 06:45:05.810359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:05.808383 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" podStartSLOduration=1.808358049 podStartE2EDuration="1.808358049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:05.806685151 +0000 UTC m=+2121.359474342" watchObservedRunningTime="2026-11-01 06:45:05.808358049 +0000 UTC m=+2121.361147243" Nov 01 06:45:07.593133 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:45:07.593097 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:45:16.579059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:16.579029 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:45:16.579447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:16.579102 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:45:33.526952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:33.526918 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr"] Nov 01 06:45:33.639270 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:33.639238 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2"] Nov 01 06:45:33.769355 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:33.769323 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx"] Nov 01 06:45:33.769575 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:33.769537 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" containerID="cri-o://7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883" gracePeriod=30 Nov 01 06:45:34.646301 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:34.646255 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq"] Nov 01 06:45:34.646713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:34.646470 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" containerID="cri-o://121d7d5438e4fd927d2dd419ad7be1cb68adcdb1df5e61ad1739016ee9c3d321" gracePeriod=30 Nov 01 06:45:37.455385 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:37.455349 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 06:45:37.455727 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:37.455408 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 06:45:37.591747 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:45:37.591718 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:45:38.527842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.527788 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" podUID="5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" containerName="payload-processing" containerID="cri-o://bd5848c4546161b808a57f9b9f9cc7a99b467c7397dc27d2c73bceab7aa744c8" gracePeriod=25 Nov 01 06:45:38.639723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.639695 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" podUID="2bd56e7d-7b78-4240-b454-b20d66bed003" containerName="payload-pre-processing" containerID="cri-o://5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488" gracePeriod=25 Nov 01 06:45:38.761206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.761173 4914 generic.go:356] "Generic (PLEG): container finished" podID="5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" containerID="bd5848c4546161b808a57f9b9f9cc7a99b467c7397dc27d2c73bceab7aa744c8" exitCode=0 Nov 01 06:45:38.761774 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.761297 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" event={"ID":"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc","Type":"ContainerDied","Data":"bd5848c4546161b808a57f9b9f9cc7a99b467c7397dc27d2c73bceab7aa744c8"} Nov 01 06:45:38.870495 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.870465 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:45:38.921461 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.921428 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:45:38.959406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.959376 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume\") pod \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " Nov 01 06:45:38.959525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.959429 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5\" (UniqueName: \"kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5\") pod \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\" (UID: \"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc\") " Nov 01 06:45:38.959614 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.959553 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume\") pod \"2bd56e7d-7b78-4240-b454-b20d66bed003\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " Nov 01 06:45:38.959672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.959638 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q\" (UniqueName: \"kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q\") pod \"2bd56e7d-7b78-4240-b454-b20d66bed003\" (UID: \"2bd56e7d-7b78-4240-b454-b20d66bed003\") " Nov 01 06:45:38.960637 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.960608 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume" pod "5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" (UID: "5defd105-0728-4d9b-8fa9-9e3e9b01f1bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:38.961047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.961023 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume" pod "2bd56e7d-7b78-4240-b454-b20d66bed003" (UID: "2bd56e7d-7b78-4240-b454-b20d66bed003"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:38.962915 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.962886 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q" pod "2bd56e7d-7b78-4240-b454-b20d66bed003" (UID: "2bd56e7d-7b78-4240-b454-b20d66bed003"). InnerVolumeSpecName "kube-api-access-7kw6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:38.963945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:38.963921 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5" pod "5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" (UID: "5defd105-0728-4d9b-8fa9-9e3e9b01f1bc"). InnerVolumeSpecName "kube-api-access-s2wx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:39.017968 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.017944 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:45:39.060590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060523 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060571 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060640 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060674 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060706 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060733 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060815 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060899 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.060979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060922 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.060979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.060956 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket\") pod \"371f0896-9ad1-440c-b439-4ed59b3144bf\" (UID: \"371f0896-9ad1-440c-b439-4ed59b3144bf\") " Nov 01 06:45:39.061161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061051 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.061208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061081 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:39.061350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061327 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.061489 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061470 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f0896-9ad1-440c-b439-4ed59b3144bf-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061489 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061488 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061484 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061500 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bd56e7d-7b78-4240-b454-b20d66bed003-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061514 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kw6q\" (UniqueName: \"kubernetes.io/projected/2bd56e7d-7b78-4240-b454-b20d66bed003-kube-api-access-7kw6q\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061528 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061536 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061546 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wx5\" (UniqueName: \"kubernetes.io/projected/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc-kube-api-access-s2wx5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.061602 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.061554 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.062763 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.062742 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:39.063215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.063197 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "kube-api-access-z5dsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:39.063526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.063499 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.063526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.063503 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo" pod "371f0896-9ad1-440c-b439-4ed59b3144bf" (UID: "371f0896-9ad1-440c-b439-4ed59b3144bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:45:39.162315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.162290 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.162315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.162317 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dsx\" (UniqueName: \"kubernetes.io/projected/371f0896-9ad1-440c-b439-4ed59b3144bf-kube-api-access-z5dsx\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.162462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.162326 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.162462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.162337 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.162462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.162345 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f0896-9ad1-440c-b439-4ed59b3144bf-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.769068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.769044 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" Nov 01 06:45:39.769412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.769084 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr" event={"ID":"5defd105-0728-4d9b-8fa9-9e3e9b01f1bc","Type":"ContainerDied","Data":"d004f0cd12c2d188436d0337c5c8f5b02f12c2fef2b0429019898a12a8b9d7ca"} Nov 01 06:45:39.769412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.769156 4914 scope.go:122] "RemoveContainer" containerID="bd5848c4546161b808a57f9b9f9cc7a99b467c7397dc27d2c73bceab7aa744c8" Nov 01 06:45:39.771047 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.770996 4914 generic.go:356] "Generic (PLEG): container finished" podID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerID="7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883" exitCode=0 Nov 01 06:45:39.771137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.771060 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" Nov 01 06:45:39.771202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.771031 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" event={"ID":"371f0896-9ad1-440c-b439-4ed59b3144bf","Type":"ContainerDied","Data":"7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883"} Nov 01 06:45:39.771259 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.771210 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx" event={"ID":"371f0896-9ad1-440c-b439-4ed59b3144bf","Type":"ContainerDied","Data":"5d59e88f8df034c40021c35e103aa51609b707d87745fb2124167fc81508f2ee"} Nov 01 06:45:39.773575 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.773556 4914 generic.go:356] "Generic (PLEG): container finished" podID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerID="121d7d5438e4fd927d2dd419ad7be1cb68adcdb1df5e61ad1739016ee9c3d321" exitCode=0 Nov 01 06:45:39.773682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.773626 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" event={"ID":"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3","Type":"ContainerDied","Data":"121d7d5438e4fd927d2dd419ad7be1cb68adcdb1df5e61ad1739016ee9c3d321"} Nov 01 06:45:39.778924 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.778903 4914 generic.go:356] "Generic (PLEG): container finished" podID="2bd56e7d-7b78-4240-b454-b20d66bed003" containerID="5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488" exitCode=0 Nov 01 06:45:39.779013 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.778952 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" Nov 01 06:45:39.779066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.778953 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" event={"ID":"2bd56e7d-7b78-4240-b454-b20d66bed003","Type":"ContainerDied","Data":"5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488"} Nov 01 06:45:39.779119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.779079 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2" event={"ID":"2bd56e7d-7b78-4240-b454-b20d66bed003","Type":"ContainerDied","Data":"6d340dada4eacbc03f39705be44c5b67551a6bd38350f8d9d11975787ab82fbb"} Nov 01 06:45:39.780233 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.780190 4914 scope.go:122] "RemoveContainer" containerID="7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883" Nov 01 06:45:39.789985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.789971 4914 scope.go:122] "RemoveContainer" containerID="7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883" Nov 01 06:45:39.790366 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:45:39.790346 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883\": container with ID starting with 7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883 not found: ID does not exist" containerID="7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883" Nov 01 06:45:39.790420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.790379 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883"} err="failed to get container status \"7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883\": rpc error: code = NotFound desc = could not find container \"7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883\": container with ID starting with 7775e75ac6e2ba14f63db2710a3cc1a4026e897e9cfb8276217faef307502883 not found: ID does not exist" Nov 01 06:45:39.790420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.790406 4914 scope.go:122] "RemoveContainer" containerID="72f7c099b07cb14a52a0b3e4cf8e36a986156487a64d598215e821e19209cba5" Nov 01 06:45:39.803326 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.803309 4914 scope.go:122] "RemoveContainer" containerID="5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488" Nov 01 06:45:39.811435 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.811416 4914 scope.go:122] "RemoveContainer" containerID="5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488" Nov 01 06:45:39.811710 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:45:39.811676 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488\": container with ID starting with 5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488 not found: ID does not exist" containerID="5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488" Nov 01 06:45:39.811767 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.811710 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488"} err="failed to get container status \"5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488\": rpc error: code = NotFound desc = could not find container \"5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488\": container with ID starting with 5a567922ac485f8337cdf33f971a4e59228d743cdd7a418a1ac8284200bf6488 not found: ID does not exist" Nov 01 06:45:39.915149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.915123 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:45:39.932159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.932133 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr"] Nov 01 06:45:39.934861 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.934839 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-030936cd-5878568bf9-zblpr"] Nov 01 06:45:39.980229 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980203 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980237 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980315 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980438 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980405 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980477 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980445 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980466 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.980513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980479 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980516 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980654 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980594 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.980687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980657 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.980718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980661 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.980718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980702 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp\") pod \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\" (UID: \"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3\") " Nov 01 06:45:39.981102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.980930 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:39.981102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.981072 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:45:39.981299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.981253 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.981299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.981294 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.981396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.981309 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.981396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.981323 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.981396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.981338 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:39.982837 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.982817 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:39.982919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.982877 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "kube-api-access-xx7hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:45:39.983101 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.983087 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 06:45:39.983137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:39.983105 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy" pod "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" (UID: "ace0ebde-a5f8-45e1-ac9e-769a7a88daf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 06:45:40.048780 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.048757 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx"] Nov 01 06:45:40.053124 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.053102 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-5lxdx"] Nov 01 06:45:40.082666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.082607 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:40.082666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.082637 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx7hp\" (UniqueName: \"kubernetes.io/projected/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-kube-api-access-xx7hp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:40.082666 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.082653 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:40.082820 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.082673 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:45:40.159984 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.159962 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2"] Nov 01 06:45:40.164165 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.164144 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-030936cd-7f9567b94c-cnzt2"] Nov 01 06:45:40.624179 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.624145 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd56e7d-7b78-4240-b454-b20d66bed003" path="/var/lib/kubelet/pods/2bd56e7d-7b78-4240-b454-b20d66bed003/volumes" Nov 01 06:45:40.624596 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.624583 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" path="/var/lib/kubelet/pods/371f0896-9ad1-440c-b439-4ed59b3144bf/volumes" Nov 01 06:45:40.624941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.624928 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" path="/var/lib/kubelet/pods/5defd105-0728-4d9b-8fa9-9e3e9b01f1bc/volumes" Nov 01 06:45:40.787159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.787134 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" Nov 01 06:45:40.787553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.787134 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq" event={"ID":"ace0ebde-a5f8-45e1-ac9e-769a7a88daf3","Type":"ContainerDied","Data":"8e1a9c98401e8a618768dcc6aa0d27ca8deb278b1dad4cea21e3022d7954b89f"} Nov 01 06:45:40.787553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.787260 4914 scope.go:122] "RemoveContainer" containerID="121d7d5438e4fd927d2dd419ad7be1cb68adcdb1df5e61ad1739016ee9c3d321" Nov 01 06:45:40.949765 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.949702 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq"] Nov 01 06:45:40.998389 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:40.998358 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-k7sxq"] Nov 01 06:45:42.177899 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.177871 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt"] Nov 01 06:45:42.178453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178439 4914 cpu_manager.go:414] "containerMap: removing container" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" Nov 01 06:45:42.178453 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178453 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178465 4914 cpu_manager.go:414] "containerMap: removing container" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178470 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178489 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2bd56e7d-7b78-4240-b454-b20d66bed003" containerName="payload-pre-processing" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178495 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd56e7d-7b78-4240-b454-b20d66bed003" containerName="payload-pre-processing" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178509 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" containerName="payload-processing" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178513 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" containerName="payload-processing" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178520 4914 cpu_manager.go:414] "containerMap: removing container" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" Nov 01 06:45:42.178545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178524 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" Nov 01 06:45:42.178764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178599 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" Nov 01 06:45:42.178764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178608 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" containerName="istio-proxy" Nov 01 06:45:42.178764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178616 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd56e7d-7b78-4240-b454-b20d66bed003" containerName="payload-pre-processing" Nov 01 06:45:42.178764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178623 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5defd105-0728-4d9b-8fa9-9e3e9b01f1bc" containerName="payload-processing" Nov 01 06:45:42.178764 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.178630 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="371f0896-9ad1-440c-b439-4ed59b3144bf" containerName="istio-proxy" Nov 01 06:45:42.183407 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.183386 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.207034 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.207009 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9kc\" (UniqueName: \"kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc\") pod \"payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.207148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.207078 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.238020 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.237995 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt"] Nov 01 06:45:42.285112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.284783 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-47bc8f48\"" Nov 01 06:45:42.318353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.318314 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.318573 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.318468 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9kc\" (UniqueName: \"kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc\") pod \"payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.319637 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.319600 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.334560 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.334533 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-47bc8f48-dockercfg-2vs8z\"" Nov 01 06:45:42.337229 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.337203 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476"] Nov 01 06:45:42.341327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.341307 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.374522 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.374500 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9kc\" (UniqueName: \"kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc\") pod \"payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.388017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.387993 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476"] Nov 01 06:45:42.496491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.496435 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:42.520733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.520709 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8fv\" (UniqueName: \"kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv\") pod \"payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.520843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.520773 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume\") pod \"payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.615976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.615951 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace0ebde-a5f8-45e1-ac9e-769a7a88daf3" path="/var/lib/kubelet/pods/ace0ebde-a5f8-45e1-ac9e-769a7a88daf3/volumes" Nov 01 06:45:42.622161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.622137 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8fv\" (UniqueName: \"kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv\") pod \"payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.622299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.622223 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume\") pod \"payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.622917 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.622897 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume\") pod \"payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.678750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.678726 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8fv\" (UniqueName: \"kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv\") pod \"payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:42.865332 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.865307 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt"] Nov 01 06:45:42.866915 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:45:42.866884 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69566da2_0890_4d50_9ecc_61f4b24d5760.slice/crio-f455925005397553cbfa7c5f9feb269b993f5f53f26d2c649075320d00d087f3 WatchSource:0}: Error finding container f455925005397553cbfa7c5f9feb269b993f5f53f26d2c649075320d00d087f3: Status 404 returned error can't find the container with id f455925005397553cbfa7c5f9feb269b993f5f53f26d2c649075320d00d087f3 Nov 01 06:45:42.955575 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:42.955544 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:43.325882 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.325859 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476"] Nov 01 06:45:43.326926 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:45:43.326904 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b36695f_0e08_451d_9a9b_36e811b788db.slice/crio-20a6981c8f953ca7af2cc13b63810b02875b5e6a9848bba1fde30e3c46a77bca WatchSource:0}: Error finding container 20a6981c8f953ca7af2cc13b63810b02875b5e6a9848bba1fde30e3c46a77bca: Status 404 returned error can't find the container with id 20a6981c8f953ca7af2cc13b63810b02875b5e6a9848bba1fde30e3c46a77bca Nov 01 06:45:43.806598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.806564 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" event={"ID":"3b36695f-0e08-451d-9a9b-36e811b788db","Type":"ContainerStarted","Data":"cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85"} Nov 01 06:45:43.806770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.806623 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:43.806770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.806643 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" event={"ID":"3b36695f-0e08-451d-9a9b-36e811b788db","Type":"ContainerStarted","Data":"20a6981c8f953ca7af2cc13b63810b02875b5e6a9848bba1fde30e3c46a77bca"} Nov 01 06:45:43.808035 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.808013 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" event={"ID":"69566da2-0890-4d50-9ecc-61f4b24d5760","Type":"ContainerStarted","Data":"973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0"} Nov 01 06:45:43.808141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.808051 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" event={"ID":"69566da2-0890-4d50-9ecc-61f4b24d5760","Type":"ContainerStarted","Data":"f455925005397553cbfa7c5f9feb269b993f5f53f26d2c649075320d00d087f3"} Nov 01 06:45:43.808141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.808070 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:43.919187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:43.919143 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" podStartSLOduration=1.919128975 podStartE2EDuration="1.919128975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:43.917387636 +0000 UTC m=+2159.470176831" watchObservedRunningTime="2026-11-01 06:45:43.919128975 +0000 UTC m=+2159.471918211" Nov 01 06:45:53.755869 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.755824 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" podStartSLOduration=11.755810773 podStartE2EDuration="11.755810773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:44.044694144 +0000 UTC m=+2159.597483340" watchObservedRunningTime="2026-11-01 06:45:53.755810773 +0000 UTC m=+2169.308600012" Nov 01 06:45:53.757215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.757195 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h"] Nov 01 06:45:53.762647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.762632 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.814849 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.814823 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8a9760-serving-cert\"" Nov 01 06:45:53.820765 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.820736 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgff\" (UniqueName: \"kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.820906 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.820809 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.820968 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.820911 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.821041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.821016 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.825509 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.825486 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h"] Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.922680 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.922737 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.922815 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.922939 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgff\" (UniqueName: \"kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.924352 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:45:53.924467 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-serving-cert: secret "maas-api-e2e-api-a-8a9760-serving-cert" not found Nov 01 06:45:53.924606 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:45:53.924538 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls podName:b8c10ea8-b0e1-4814-84c8-dc464cf4f869 nodeName:}" failed. No retries permitted until 2026-11-01 06:45:54.424519978 +0000 UTC m=+2169.977309152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls") pod "maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" (UID: "b8c10ea8-b0e1-4814-84c8-dc464cf4f869") : secret "maas-api-e2e-api-a-8a9760-serving-cert" not found Nov 01 06:45:53.932323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.932300 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:53.979196 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:53.979176 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgff\" (UniqueName: \"kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:54.427719 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:54.427687 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:54.430131 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:54.430101 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\") pod \"maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:54.674692 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:54.674667 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:54.816212 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:54.816181 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:45:54.816639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:54.816254 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:45:55.095475 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:55.095447 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h"] Nov 01 06:45:55.096965 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:45:55.096925 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c10ea8_b0e1_4814_84c8_dc464cf4f869.slice/crio-2ce0781f118a69da10fecc634626534df28874bbb84bcbf9b88548fa1db29150 WatchSource:0}: Error finding container 2ce0781f118a69da10fecc634626534df28874bbb84bcbf9b88548fa1db29150: Status 404 returned error can't find the container with id 2ce0781f118a69da10fecc634626534df28874bbb84bcbf9b88548fa1db29150 Nov 01 06:45:55.098336 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:55.098315 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:45:55.879332 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:55.879302 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" event={"ID":"b8c10ea8-b0e1-4814-84c8-dc464cf4f869","Type":"ContainerStarted","Data":"2ce0781f118a69da10fecc634626534df28874bbb84bcbf9b88548fa1db29150"} Nov 01 06:45:56.887711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:56.887675 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" event={"ID":"b8c10ea8-b0e1-4814-84c8-dc464cf4f869","Type":"ContainerStarted","Data":"cb9a8ac22b70e03f0ebf095826647bad35eb1657e51c1ba4c22ce728ca3d2c90"} Nov 01 06:45:56.888152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:56.887748 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:45:57.003291 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:45:57.003235 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" podStartSLOduration=3.1021126 podStartE2EDuration="4.003221785s" totalImagesPullingTime="901.109185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:45:57.001793926 +0000 UTC m=+2172.554583118" watchObservedRunningTime="2026-11-01 06:45:57.003221785 +0000 UTC m=+2172.556010975" Nov 01 06:46:02.900041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:02.900014 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:46:07.591654 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:46:07.591620 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:46:24.593583 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.593547 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9"] Nov 01 06:46:24.597398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.597382 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.651539 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.651513 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-caf74d\"" Nov 01 06:46:24.705806 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.704637 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-caf74d-dockercfg-vs92c\"" Nov 01 06:46:24.710258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.710233 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9"] Nov 01 06:46:24.757999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.757967 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq"] Nov 01 06:46:24.762201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.762176 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:24.786893 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.786866 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bnr\" (UniqueName: \"kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr\") pod \"payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.787017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.786952 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.812732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.812707 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq"] Nov 01 06:46:24.887566 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.887535 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:24.887709 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.887599 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.887709 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.887656 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtm44\" (UniqueName: \"kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44\") pod \"payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:24.887875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.887851 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bnr\" (UniqueName: \"kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr\") pod \"payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.888264 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.888242 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.944395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.944371 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bnr\" (UniqueName: \"kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr\") pod \"payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:24.988919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.988897 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:24.989050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.988940 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtm44\" (UniqueName: \"kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44\") pod \"payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:24.989611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:24.989589 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:25.045054 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:25.045029 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtm44\" (UniqueName: \"kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44\") pod \"payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:25.078165 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:25.078142 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:25.210652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:25.210580 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:25.448787 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:25.448765 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq"] Nov 01 06:46:25.449857 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:46:25.449835 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b5ec26_260b_4fa2_a798_a6bdb60a69a5.slice/crio-979cdba9182f4d061af4d856e6d936c1f2adc362b103d2121d682693f385e6eb WatchSource:0}: Error finding container 979cdba9182f4d061af4d856e6d936c1f2adc362b103d2121d682693f385e6eb: Status 404 returned error can't find the container with id 979cdba9182f4d061af4d856e6d936c1f2adc362b103d2121d682693f385e6eb Nov 01 06:46:25.581350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:25.581328 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9"] Nov 01 06:46:25.582160 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:46:25.582139 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4daf58c_f842_4196_bdc0_6b180aa5a781.slice/crio-2a87c99676be78db80c8021d85d7a863bd79c2057e6f2214d86d85bad506cdc4 WatchSource:0}: Error finding container 2a87c99676be78db80c8021d85d7a863bd79c2057e6f2214d86d85bad506cdc4: Status 404 returned error can't find the container with id 2a87c99676be78db80c8021d85d7a863bd79c2057e6f2214d86d85bad506cdc4 Nov 01 06:46:26.073206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.073135 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" event={"ID":"c4daf58c-f842-4196-bdc0-6b180aa5a781","Type":"ContainerStarted","Data":"cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d"} Nov 01 06:46:26.073206 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.073192 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" event={"ID":"c4daf58c-f842-4196-bdc0-6b180aa5a781","Type":"ContainerStarted","Data":"2a87c99676be78db80c8021d85d7a863bd79c2057e6f2214d86d85bad506cdc4"} Nov 01 06:46:26.073611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.073224 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:26.074675 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.074655 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" event={"ID":"93b5ec26-260b-4fa2-a798-a6bdb60a69a5","Type":"ContainerStarted","Data":"41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e"} Nov 01 06:46:26.074767 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.074688 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" event={"ID":"93b5ec26-260b-4fa2-a798-a6bdb60a69a5","Type":"ContainerStarted","Data":"979cdba9182f4d061af4d856e6d936c1f2adc362b103d2121d682693f385e6eb"} Nov 01 06:46:26.074813 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.074774 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:26.187974 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.187928 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" podStartSLOduration=2.187914871 podStartE2EDuration="2.187914871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:26.185674701 +0000 UTC m=+2201.738463894" watchObservedRunningTime="2026-11-01 06:46:26.187914871 +0000 UTC m=+2201.740704062" Nov 01 06:46:26.316346 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:26.309628 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" podStartSLOduration=2.309607713 podStartE2EDuration="2.309607713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:26.307702118 +0000 UTC m=+2201.860491312" watchObservedRunningTime="2026-11-01 06:46:26.309607713 +0000 UTC m=+2201.862396906" Nov 01 06:46:33.498986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.498954 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh"] Nov 01 06:46:33.504165 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.504146 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.598431 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.598387 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.598581 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.598444 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmq58\" (UniqueName: \"kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.598642 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.598588 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.598781 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.598767 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.607424 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.607396 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-47bc8f48-serving-cert\"" Nov 01 06:46:33.610028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.610008 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh"] Nov 01 06:46:33.699546 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.699517 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.699686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.699561 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.699686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.699582 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmq58\" (UniqueName: \"kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.699686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.699634 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.699787 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:46:33.699732 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-serving-cert: secret "maas-api-e2e-mt-47bc8f48-serving-cert" not found Nov 01 06:46:33.699820 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:46:33.699796 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls podName:3a01b668-5a0f-4a6c-9c67-8e6161dcbc02 nodeName:}" failed. No retries permitted until 2026-11-01 06:46:34.199780056 +0000 UTC m=+2209.752569250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls") pod "maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" (UID: "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02") : secret "maas-api-e2e-mt-47bc8f48-serving-cert" not found Nov 01 06:46:33.700380 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.700352 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.702141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.702121 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:33.772397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:33.772318 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmq58\" (UniqueName: \"kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:34.209953 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:34.209917 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:34.212445 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:34.212418 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\") pod \"maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:34.424365 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:34.424336 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:34.795197 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:34.795171 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh"] Nov 01 06:46:34.796264 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:46:34.796240 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a01b668_5a0f_4a6c_9c67_8e6161dcbc02.slice/crio-44c81b33079f1bc95cfab4b161718b7d7d79cac19b9a168d319776fb7d142680 WatchSource:0}: Error finding container 44c81b33079f1bc95cfab4b161718b7d7d79cac19b9a168d319776fb7d142680: Status 404 returned error can't find the container with id 44c81b33079f1bc95cfab4b161718b7d7d79cac19b9a168d319776fb7d142680 Nov 01 06:46:35.128545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:35.128519 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" event={"ID":"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02","Type":"ContainerStarted","Data":"44c81b33079f1bc95cfab4b161718b7d7d79cac19b9a168d319776fb7d142680"} Nov 01 06:46:36.136171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:36.136133 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" event={"ID":"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02","Type":"ContainerStarted","Data":"df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda"} Nov 01 06:46:36.136589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:36.136202 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:46:36.249659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:36.249606 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" podStartSLOduration=2.422252962 podStartE2EDuration="3.249587428s" totalImagesPullingTime="827.334466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:46:36.248951462 +0000 UTC m=+2211.801740654" watchObservedRunningTime="2026-11-01 06:46:36.249587428 +0000 UTC m=+2211.802376620" Nov 01 06:46:37.083014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:37.082981 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 06:46:37.083181 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:37.083114 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 06:46:37.624228 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:46:37.624185 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:46:42.147418 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:46:42.147388 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:47:07.591954 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:07.591921 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:47:11.542496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.542460 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx"] Nov 01 06:47:11.546523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.546508 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.619804 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.619763 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx"] Nov 01 06:47:11.652188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.649397 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d855db\"" Nov 01 06:47:11.652188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.649595 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d855db-dockercfg-sfvmq\"" Nov 01 06:47:11.653180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.653142 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2l7\" (UniqueName: \"kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7\") pod \"payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.653572 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.653541 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.754739 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.754700 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.754925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.754755 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2l7\" (UniqueName: \"kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7\") pod \"payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.755481 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.755457 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.850347 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.848559 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2l7\" (UniqueName: \"kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7\") pod \"payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:11.860559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:11.860526 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:12.241671 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:12.241640 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx"] Nov 01 06:47:12.243309 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:47:12.243256 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d99c13b_daec_4f3c_9a38_4e13a6c1eee4.slice/crio-6267da35056d20984b7469e876ae4903bf3e11de61c08d647f05b848b23d1312 WatchSource:0}: Error finding container 6267da35056d20984b7469e876ae4903bf3e11de61c08d647f05b848b23d1312: Status 404 returned error can't find the container with id 6267da35056d20984b7469e876ae4903bf3e11de61c08d647f05b848b23d1312 Nov 01 06:47:12.353360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:12.353326 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" event={"ID":"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4","Type":"ContainerStarted","Data":"8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038"} Nov 01 06:47:12.353483 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:12.353390 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" event={"ID":"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4","Type":"ContainerStarted","Data":"6267da35056d20984b7469e876ae4903bf3e11de61c08d647f05b848b23d1312"} Nov 01 06:47:12.353558 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:12.353539 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:12.468350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:12.468306 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" podStartSLOduration=1.468291344 podStartE2EDuration="1.468291344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:12.46640665 +0000 UTC m=+2248.019195854" watchObservedRunningTime="2026-11-01 06:47:12.468291344 +0000 UTC m=+2248.021080539" Nov 01 06:47:23.369245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:23.369213 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:47:32.081349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.081316 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc"] Nov 01 06:47:32.085833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.085811 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.181810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.181775 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.181964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.181859 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.181964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.181920 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt64d\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.182101 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.182082 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.194738 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194708 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-f6f2772a70c5\"" Nov 01 06:47:32.194878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194781 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-294b20a2cf15\"" Nov 01 06:47:32.194878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194796 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25719d\"" Nov 01 06:47:32.194878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194801 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"routing-overlay\"" Nov 01 06:47:32.194878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194781 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"agc-cred-d41a0e39961b\"" Nov 01 06:47:32.194878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194826 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"payload-processing-external-model-e2e--bf6d8971-dockercfg-6qmzh\"" Nov 01 06:47:32.194878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194805 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"openshift-service-ca.crt\"" Nov 01 06:47:32.195103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.194960 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25719d\"/\"kube-root-ca.crt\"" Nov 01 06:47:32.197674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.197656 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc"] Nov 01 06:47:32.283108 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.283072 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.283303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.283121 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.283303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.283160 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt64d\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.283303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.283231 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.283879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.283858 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.283953 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.283921 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.285810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.285789 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.340056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.339988 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt64d\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.425486 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.425454 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:32.801320 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:47:32.801262 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260f4c99_9b55_4530_9d98_e3342d64f5f5.slice/crio-45d60149c0ea09364cd092f59e50bfce5000c5310cbf5c4e28e0ae498d8af65a WatchSource:0}: Error finding container 45d60149c0ea09364cd092f59e50bfce5000c5310cbf5c4e28e0ae498d8af65a: Status 404 returned error can't find the container with id 45d60149c0ea09364cd092f59e50bfce5000c5310cbf5c4e28e0ae498d8af65a Nov 01 06:47:32.802227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:32.802192 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc"] Nov 01 06:47:33.505214 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:33.505182 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" event={"ID":"260f4c99-9b55-4530-9d98-e3342d64f5f5","Type":"ContainerStarted","Data":"6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636"} Nov 01 06:47:33.505648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:33.505388 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:33.505648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:33.505423 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" event={"ID":"260f4c99-9b55-4530-9d98-e3342d64f5f5","Type":"ContainerStarted","Data":"45d60149c0ea09364cd092f59e50bfce5000c5310cbf5c4e28e0ae498d8af65a"} Nov 01 06:47:33.618824 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:33.618774 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" podStartSLOduration=1.618761923 podStartE2EDuration="1.618761923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:33.617701334 +0000 UTC m=+2269.170490527" watchObservedRunningTime="2026-11-01 06:47:33.618761923 +0000 UTC m=+2269.171551116" Nov 01 06:47:34.593352 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.593316 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb"] Nov 01 06:47:34.597568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.597548 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.607619 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.607580 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.607755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.607740 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.607832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.607784 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.607922 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.607904 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqcz\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.700788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700750 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-529628bc0fc6\"" Nov 01 06:47:34.700788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700778 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-97c3302b7889\"" Nov 01 06:47:34.700788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700765 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"openshift-service-ca.crt\"" Nov 01 06:47:34.701066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700759 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ff1dc\"" Nov 01 06:47:34.701066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700839 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"kube-root-ca.crt\"" Nov 01 06:47:34.701066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700850 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-e2e--528f7081-dockercfg-c6b6l\"" Nov 01 06:47:34.701066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700910 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"routing-overlay\"" Nov 01 06:47:34.701066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.700772 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-841de3b011bd\"" Nov 01 06:47:34.703741 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.703724 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb"] Nov 01 06:47:34.708635 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.708612 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.708714 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.708657 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.708762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.708747 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqcz\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.708832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.708815 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.709491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.709418 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.709491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.709450 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.711251 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.711231 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.765102 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.765078 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqcz\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:34.911833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:34.911806 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:35.285173 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:35.285138 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb"] Nov 01 06:47:35.286163 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:47:35.286139 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6b0a87_93d6_4042_b667_778f316b04bf.slice/crio-4f865d66a856c486940ca8b9f034dbc7c9607c39e1c0084e292af5f58fc9555a WatchSource:0}: Error finding container 4f865d66a856c486940ca8b9f034dbc7c9607c39e1c0084e292af5f58fc9555a: Status 404 returned error can't find the container with id 4f865d66a856c486940ca8b9f034dbc7c9607c39e1c0084e292af5f58fc9555a Nov 01 06:47:35.518247 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:35.518209 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" event={"ID":"8e6b0a87-93d6-4042-b667-778f316b04bf","Type":"ContainerStarted","Data":"fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41"} Nov 01 06:47:35.518424 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:35.518267 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" event={"ID":"8e6b0a87-93d6-4042-b667-778f316b04bf","Type":"ContainerStarted","Data":"4f865d66a856c486940ca8b9f034dbc7c9607c39e1c0084e292af5f58fc9555a"} Nov 01 06:47:35.518424 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:35.518314 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:35.631570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:35.631519 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" podStartSLOduration=1.63150642 podStartE2EDuration="1.63150642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:35.630329571 +0000 UTC m=+2271.183118764" watchObservedRunningTime="2026-11-01 06:47:35.63150642 +0000 UTC m=+2271.184295613" Nov 01 06:47:36.634406 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:36.634373 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:36.634406 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:36.634403 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:36.634887 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:36.634472 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials podName:8e6b0a87-93d6-4042-b667-778f316b04bf nodeName:}" failed. No retries permitted until 2026-11-01 06:47:37.134450986 +0000 UTC m=+2272.687240157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf") : secret "agc-cred-841de3b011bd" not found Nov 01 06:47:37.121916 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.121805 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc"] Nov 01 06:47:37.134975 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.134919 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" podUID="260f4c99-9b55-4530-9d98-e3342d64f5f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:37.143503 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:37.143475 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:37.143503 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:37.143501 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:37.143715 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:37.143560 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials podName:8e6b0a87-93d6-4042-b667-778f316b04bf nodeName:}" failed. No retries permitted until 2026-11-01 06:47:38.143545533 +0000 UTC m=+2273.696334708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf") : secret "agc-cred-841de3b011bd" not found Nov 01 06:47:37.166004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.165978 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h"] Nov 01 06:47:37.170339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.170324 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.224588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.224564 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h"] Nov 01 06:47:37.244977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.244942 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4lc\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.245161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.244998 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.245161 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.245082 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.245313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.245167 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.346784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.346732 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.347103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.346899 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4lc\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.347103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.347032 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.347362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.347122 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.348011 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.347987 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.348377 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.348351 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.350344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.350326 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.403476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.403407 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4lc\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.484595 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.484557 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:37.595677 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:37.595637 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:47:37.858171 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:37.858145 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h"] Nov 01 06:47:37.859053 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:47:37.859028 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f291db_150a_4afb_98f4_a89003717a1b.slice/crio-33201cf36aa46c4a235d00bfd4bb7073958a204cb815047a76f6e87ca4da9036 WatchSource:0}: Error finding container 33201cf36aa46c4a235d00bfd4bb7073958a204cb815047a76f6e87ca4da9036: Status 404 returned error can't find the container with id 33201cf36aa46c4a235d00bfd4bb7073958a204cb815047a76f6e87ca4da9036 Nov 01 06:47:38.156509 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:38.156430 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:38.156509 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:38.156457 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:38.156509 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:38.156509 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials podName:8e6b0a87-93d6-4042-b667-778f316b04bf nodeName:}" failed. No retries permitted until 2026-11-01 06:47:40.1564941 +0000 UTC m=+2275.709283285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf") : secret "agc-cred-841de3b011bd" not found Nov 01 06:47:38.551667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:38.551567 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" event={"ID":"54f291db-150a-4afb-98f4-a89003717a1b","Type":"ContainerStarted","Data":"37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249"} Nov 01 06:47:38.551667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:38.551618 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" event={"ID":"54f291db-150a-4afb-98f4-a89003717a1b","Type":"ContainerStarted","Data":"33201cf36aa46c4a235d00bfd4bb7073958a204cb815047a76f6e87ca4da9036"} Nov 01 06:47:38.551667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:38.551657 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:47:38.667373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:38.667316 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" podStartSLOduration=1.6673013650000001 podStartE2EDuration="1.667301365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:47:38.663824613 +0000 UTC m=+2274.216613805" watchObservedRunningTime="2026-11-01 06:47:38.667301365 +0000 UTC m=+2274.220090557" Nov 01 06:47:39.612969 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:39.612934 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb"] Nov 01 06:47:40.177649 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:40.177618 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:40.177649 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:40.177645 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:40.177853 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:40.177699 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials podName:8e6b0a87-93d6-4042-b667-778f316b04bf nodeName:}" failed. No retries permitted until 2026-11-01 06:47:44.17768582 +0000 UTC m=+2279.730474992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf") : secret "agc-cred-841de3b011bd" not found Nov 01 06:47:42.123551 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.123504 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" podUID="260f4c99-9b55-4530-9d98-e3342d64f5f5" containerName="payload-processing" containerID="cri-o://6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636" gracePeriod=25 Nov 01 06:47:42.372330 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.372304 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:42.408425 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.408369 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d\") pod \"260f4c99-9b55-4530-9d98-e3342d64f5f5\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " Nov 01 06:47:42.408550 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.408424 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume\") pod \"260f4c99-9b55-4530-9d98-e3342d64f5f5\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " Nov 01 06:47:42.408550 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.408497 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials\") pod \"260f4c99-9b55-4530-9d98-e3342d64f5f5\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " Nov 01 06:47:42.408665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.408644 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume\") pod \"260f4c99-9b55-4530-9d98-e3342d64f5f5\" (UID: \"260f4c99-9b55-4530-9d98-e3342d64f5f5\") " Nov 01 06:47:42.408874 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.408850 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume" pod "260f4c99-9b55-4530-9d98-e3342d64f5f5" (UID: "260f4c99-9b55-4530-9d98-e3342d64f5f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:47:42.408971 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.408951 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume" pod "260f4c99-9b55-4530-9d98-e3342d64f5f5" (UID: "260f4c99-9b55-4530-9d98-e3342d64f5f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:47:42.409083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.409068 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:42.409124 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.409086 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/260f4c99-9b55-4530-9d98-e3342d64f5f5-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:42.410633 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.410615 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d" pod "260f4c99-9b55-4530-9d98-e3342d64f5f5" (UID: "260f4c99-9b55-4530-9d98-e3342d64f5f5"). InnerVolumeSpecName "kube-api-access-wt64d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:47:42.410789 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.410774 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials" pod "260f4c99-9b55-4530-9d98-e3342d64f5f5" (UID: "260f4c99-9b55-4530-9d98-e3342d64f5f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:47:42.510685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.510650 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt64d\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-kube-api-access-wt64d\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:42.510685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.510685 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/260f4c99-9b55-4530-9d98-e3342d64f5f5-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:42.575577 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.575553 4914 generic.go:356] "Generic (PLEG): container finished" podID="260f4c99-9b55-4530-9d98-e3342d64f5f5" containerID="6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636" exitCode=0 Nov 01 06:47:42.575706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.575603 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" Nov 01 06:47:42.575706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.575642 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" event={"ID":"260f4c99-9b55-4530-9d98-e3342d64f5f5","Type":"ContainerDied","Data":"6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636"} Nov 01 06:47:42.575811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.575704 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc" event={"ID":"260f4c99-9b55-4530-9d98-e3342d64f5f5","Type":"ContainerDied","Data":"45d60149c0ea09364cd092f59e50bfce5000c5310cbf5c4e28e0ae498d8af65a"} Nov 01 06:47:42.575811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.575740 4914 scope.go:122] "RemoveContainer" containerID="6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636" Nov 01 06:47:42.585574 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.585558 4914 scope.go:122] "RemoveContainer" containerID="6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636" Nov 01 06:47:42.585924 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:42.585892 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636\": container with ID starting with 6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636 not found: ID does not exist" containerID="6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636" Nov 01 06:47:42.586018 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.585929 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636"} err="failed to get container status \"6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636\": rpc error: code = NotFound desc = could not find container \"6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636\": container with ID starting with 6bf764c2ff0204cc7b6186fd83639aa4c01537670647c66dc97e8b7f7d07b636 not found: ID does not exist" Nov 01 06:47:42.738991 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.738924 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc"] Nov 01 06:47:42.741450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:42.741432 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-75c896sdc"] Nov 01 06:47:44.228673 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:44.228647 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1ff1dc/agc-cred-841de3b011bd: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:44.228673 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:44.228672 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb: secret "agc-cred-841de3b011bd" not found Nov 01 06:47:44.229084 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:44.228725 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials podName:8e6b0a87-93d6-4042-b667-778f316b04bf nodeName:}" failed. No retries permitted until 2026-11-01 06:47:52.228710462 +0000 UTC m=+2287.781499634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf") : secret "agc-cred-841de3b011bd" not found Nov 01 06:47:44.613179 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.613117 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" podUID="8e6b0a87-93d6-4042-b667-778f316b04bf" containerName="payload-processing" containerID="cri-o://fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41" gracePeriod=25 Nov 01 06:47:44.616392 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.616371 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260f4c99-9b55-4530-9d98-e3342d64f5f5" path="/var/lib/kubelet/pods/260f4c99-9b55-4530-9d98-e3342d64f5f5/volumes" Nov 01 06:47:44.876992 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.876939 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:44.934975 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.934951 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials\") pod \"8e6b0a87-93d6-4042-b667-778f316b04bf\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " Nov 01 06:47:44.935092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.935012 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume\") pod \"8e6b0a87-93d6-4042-b667-778f316b04bf\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " Nov 01 06:47:44.935092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.935057 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume\") pod \"8e6b0a87-93d6-4042-b667-778f316b04bf\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " Nov 01 06:47:44.935092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.935073 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz\") pod \"8e6b0a87-93d6-4042-b667-778f316b04bf\" (UID: \"8e6b0a87-93d6-4042-b667-778f316b04bf\") " Nov 01 06:47:44.935400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.935375 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume" pod "8e6b0a87-93d6-4042-b667-778f316b04bf" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:47:44.935517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.935421 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume" pod "8e6b0a87-93d6-4042-b667-778f316b04bf" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:47:44.937224 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.937197 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz" pod "8e6b0a87-93d6-4042-b667-778f316b04bf" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf"). InnerVolumeSpecName "kube-api-access-2mqcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:47:44.937317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:44.937287 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials" pod "8e6b0a87-93d6-4042-b667-778f316b04bf" (UID: "8e6b0a87-93d6-4042-b667-778f316b04bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:47:45.036726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.036699 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:45.036726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.036726 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mqcz\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-kube-api-access-2mqcz\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:45.036906 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.036735 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e6b0a87-93d6-4042-b667-778f316b04bf-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:45.036906 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.036743 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6b0a87-93d6-4042-b667-778f316b04bf-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:47:45.594880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.594799 4914 generic.go:356] "Generic (PLEG): container finished" podID="8e6b0a87-93d6-4042-b667-778f316b04bf" containerID="fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41" exitCode=0 Nov 01 06:47:45.594880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.594859 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" Nov 01 06:47:45.595413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.594886 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" event={"ID":"8e6b0a87-93d6-4042-b667-778f316b04bf","Type":"ContainerDied","Data":"fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41"} Nov 01 06:47:45.595413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.594975 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb" event={"ID":"8e6b0a87-93d6-4042-b667-778f316b04bf","Type":"ContainerDied","Data":"4f865d66a856c486940ca8b9f034dbc7c9607c39e1c0084e292af5f58fc9555a"} Nov 01 06:47:45.595413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.595009 4914 scope.go:122] "RemoveContainer" containerID="fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41" Nov 01 06:47:45.605210 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.605192 4914 scope.go:122] "RemoveContainer" containerID="fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41" Nov 01 06:47:45.605484 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:47:45.605463 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41\": container with ID starting with fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41 not found: ID does not exist" containerID="fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41" Nov 01 06:47:45.605556 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.605496 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41"} err="failed to get container status \"fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41\": rpc error: code = NotFound desc = could not find container \"fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41\": container with ID starting with fefcaf45c383964dd59549454892363d20d23d730d2d9f63929eb774c6279d41 not found: ID does not exist" Nov 01 06:47:45.758655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.758621 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb"] Nov 01 06:47:45.808872 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:45.808836 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-7f95b6mzb"] Nov 01 06:47:46.616960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:46.616929 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6b0a87-93d6-4042-b667-778f316b04bf" path="/var/lib/kubelet/pods/8e6b0a87-93d6-4042-b667-778f316b04bf/volumes" Nov 01 06:47:49.560399 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:49.560354 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:47:59.558517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:47:59.558441 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:02.828729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.828696 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h"] Nov 01 06:48:02.830121 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.830091 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:02.859397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.859371 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p"] Nov 01 06:48:02.860018 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.860003 4914 cpu_manager.go:414] "containerMap: removing container" podUID="260f4c99-9b55-4530-9d98-e3342d64f5f5" containerName="payload-processing" Nov 01 06:48:02.860074 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.860019 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260f4c99-9b55-4530-9d98-e3342d64f5f5" containerName="payload-processing" Nov 01 06:48:02.860074 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.860048 4914 cpu_manager.go:414] "containerMap: removing container" podUID="8e6b0a87-93d6-4042-b667-778f316b04bf" containerName="payload-processing" Nov 01 06:48:02.860074 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.860053 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6b0a87-93d6-4042-b667-778f316b04bf" containerName="payload-processing" Nov 01 06:48:02.860166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.860160 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="260f4c99-9b55-4530-9d98-e3342d64f5f5" containerName="payload-processing" Nov 01 06:48:02.860198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.860175 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6b0a87-93d6-4042-b667-778f316b04bf" containerName="payload-processing" Nov 01 06:48:02.865244 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.865226 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.866784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.866755 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.866921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.866790 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.866921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.866884 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqr9z\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.867083 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.866939 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.920535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.920504 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p"] Nov 01 06:48:02.967626 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.967600 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.967753 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.967705 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.967753 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.967729 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.967832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.967764 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqr9z\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.968465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.968439 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.968530 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.968513 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:02.970287 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:02.970257 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:03.037398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.037362 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqr9z\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z\") pod \"payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:03.180068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.180035 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:03.552868 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.552839 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p"] Nov 01 06:48:03.553462 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:03.553439 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056002ce_fcc2_49a5_aea3_96592c8d4e58.slice/crio-a2bb551b0760dd98e9723358bde05a33dc65769efa816794cc05688c64641c21 WatchSource:0}: Error finding container a2bb551b0760dd98e9723358bde05a33dc65769efa816794cc05688c64641c21: Status 404 returned error can't find the container with id a2bb551b0760dd98e9723358bde05a33dc65769efa816794cc05688c64641c21 Nov 01 06:48:03.701269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.701237 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" event={"ID":"056002ce-fcc2-49a5-aea3-96592c8d4e58","Type":"ContainerStarted","Data":"8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f"} Nov 01 06:48:03.701417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.701299 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" event={"ID":"056002ce-fcc2-49a5-aea3-96592c8d4e58","Type":"ContainerStarted","Data":"a2bb551b0760dd98e9723358bde05a33dc65769efa816794cc05688c64641c21"} Nov 01 06:48:03.701417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.701323 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:03.815209 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:03.815128 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" podStartSLOduration=1.8151139920000001 podStartE2EDuration="1.815113992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:03.813759578 +0000 UTC m=+2299.366548763" watchObservedRunningTime="2026-11-01 06:48:03.815113992 +0000 UTC m=+2299.367903212" Nov 01 06:48:05.351474 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.351440 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb"] Nov 01 06:48:05.355782 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.355767 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.393210 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.393177 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.393354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.393337 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfv9\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.393419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.393374 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.393485 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.393464 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.407858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.407839 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"agc-cred-529628bc0fc6\"" Nov 01 06:48:05.457639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.457576 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"kube-root-ca.crt\"" Nov 01 06:48:05.457639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.457576 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"openshift-service-ca.crt\"" Nov 01 06:48:05.457639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.457615 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"routing-overlay\"" Nov 01 06:48:05.457639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.457584 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-e2e--528f7081-dockercfg-c6b6l\"" Nov 01 06:48:05.457951 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.457646 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ff1dc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ff1dc\"" Nov 01 06:48:05.460448 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.460427 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb"] Nov 01 06:48:05.494902 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.494878 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.495010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.494943 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfv9\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.495010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.494966 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.495010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.494998 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.495663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.495642 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.495743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.495691 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.497459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.497443 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.551568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.551544 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfv9\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9\") pod \"payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:05.669647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:05.669592 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:06.042319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:06.042254 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb"] Nov 01 06:48:06.042697 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:06.042674 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc80f414_2593_4b63_8838_64d1f217d9bd.slice/crio-155b62dd3c8a0885d140ca0a3fb8a73093ffbb7f1bb83fbad86cc3bb910e2960 WatchSource:0}: Error finding container 155b62dd3c8a0885d140ca0a3fb8a73093ffbb7f1bb83fbad86cc3bb910e2960: Status 404 returned error can't find the container with id 155b62dd3c8a0885d140ca0a3fb8a73093ffbb7f1bb83fbad86cc3bb910e2960 Nov 01 06:48:06.719643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:06.719607 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" event={"ID":"cc80f414-2593-4b63-8838-64d1f217d9bd","Type":"ContainerStarted","Data":"9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463"} Nov 01 06:48:06.720022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:06.719660 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" event={"ID":"cc80f414-2593-4b63-8838-64d1f217d9bd","Type":"ContainerStarted","Data":"155b62dd3c8a0885d140ca0a3fb8a73093ffbb7f1bb83fbad86cc3bb910e2960"} Nov 01 06:48:06.720022 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:06.719742 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:06.834406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:06.834351 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" podStartSLOduration=1.83433513 podStartE2EDuration="1.83433513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:06.83236832 +0000 UTC m=+2302.385157514" watchObservedRunningTime="2026-11-01 06:48:06.83433513 +0000 UTC m=+2302.387124324" Nov 01 06:48:07.592623 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:48:07.592591 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:48:07.829578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:07.829541 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" containerID="cri-o://37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249" gracePeriod=25 Nov 01 06:48:08.083430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.083401 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:48:08.138233 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138210 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume\") pod \"54f291db-150a-4afb-98f4-a89003717a1b\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " Nov 01 06:48:08.138398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138328 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials\") pod \"54f291db-150a-4afb-98f4-a89003717a1b\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " Nov 01 06:48:08.138398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138385 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc\") pod \"54f291db-150a-4afb-98f4-a89003717a1b\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " Nov 01 06:48:08.138503 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138442 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume\") pod \"54f291db-150a-4afb-98f4-a89003717a1b\" (UID: \"54f291db-150a-4afb-98f4-a89003717a1b\") " Nov 01 06:48:08.138660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138637 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume" pod "54f291db-150a-4afb-98f4-a89003717a1b" (UID: "54f291db-150a-4afb-98f4-a89003717a1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:08.138820 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138801 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume" pod "54f291db-150a-4afb-98f4-a89003717a1b" (UID: "54f291db-150a-4afb-98f4-a89003717a1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:08.138890 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.138848 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:08.140535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.140512 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials" pod "54f291db-150a-4afb-98f4-a89003717a1b" (UID: "54f291db-150a-4afb-98f4-a89003717a1b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:08.140612 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.140537 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc" pod "54f291db-150a-4afb-98f4-a89003717a1b" (UID: "54f291db-150a-4afb-98f4-a89003717a1b"). InnerVolumeSpecName "kube-api-access-mp4lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:08.239659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.239628 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54f291db-150a-4afb-98f4-a89003717a1b-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:08.239659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.239654 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:08.239659 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.239664 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4lc\" (UniqueName: \"kubernetes.io/projected/54f291db-150a-4afb-98f4-a89003717a1b-kube-api-access-mp4lc\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:08.746594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.746560 4914 generic.go:356] "Generic (PLEG): container finished" podID="54f291db-150a-4afb-98f4-a89003717a1b" containerID="37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249" exitCode=0 Nov 01 06:48:08.746760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.746618 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" Nov 01 06:48:08.746760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.746651 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" event={"ID":"54f291db-150a-4afb-98f4-a89003717a1b","Type":"ContainerDied","Data":"37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249"} Nov 01 06:48:08.746760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.746706 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h" event={"ID":"54f291db-150a-4afb-98f4-a89003717a1b","Type":"ContainerDied","Data":"33201cf36aa46c4a235d00bfd4bb7073958a204cb815047a76f6e87ca4da9036"} Nov 01 06:48:08.746760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.746735 4914 scope.go:122] "RemoveContainer" containerID="37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249" Nov 01 06:48:08.756078 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.756059 4914 scope.go:122] "RemoveContainer" containerID="37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249" Nov 01 06:48:08.756377 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:48:08.756358 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249\": container with ID starting with 37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249 not found: ID does not exist" containerID="37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249" Nov 01 06:48:08.756428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.756384 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249"} err="failed to get container status \"37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249\": rpc error: code = NotFound desc = could not find container \"37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249\": container with ID starting with 37e867b86e480f4e8ce5d08807afeaa6ff6a0a9e6ad7aa27b621c29e83f5a249 not found: ID does not exist" Nov 01 06:48:08.908946 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.908861 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h"] Nov 01 06:48:08.961592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:08.961558 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-6fdb9h48h"] Nov 01 06:48:10.124369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.124338 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5"] Nov 01 06:48:10.124981 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.124967 4914 cpu_manager.go:414] "containerMap: removing container" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" Nov 01 06:48:10.125044 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.124981 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" Nov 01 06:48:10.125113 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.125104 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f291db-150a-4afb-98f4-a89003717a1b" containerName="payload-processing" Nov 01 06:48:10.128560 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.128545 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.158898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.158871 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8btr\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.159017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.158951 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.159017 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.158994 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.159103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.159019 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.180725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.180703 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-a83cd424b5a5\"" Nov 01 06:48:10.180805 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.180771 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dc6e48\"" Nov 01 06:48:10.230327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.230301 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"agc-cred-120370fe39d3\"" Nov 01 06:48:10.230327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.230302 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"routing-overlay\"" Nov 01 06:48:10.230327 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.230308 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"kube-root-ca.crt\"" Nov 01 06:48:10.230568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.230519 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"openshift-service-ca.crt\"" Nov 01 06:48:10.230609 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.230596 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dc6e48\"/\"payload-processing-external-model-e2e--958558de-dockercfg-qjzd6\"" Nov 01 06:48:10.233424 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.233403 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5"] Nov 01 06:48:10.260065 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.260039 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.260185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.260096 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.260185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.260113 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.260185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.260164 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8btr\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.260878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.260861 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.260945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.260911 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.262745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.262728 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.316824 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.316795 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8btr\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.442039 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.441982 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:10.616287 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.616247 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f291db-150a-4afb-98f4-a89003717a1b" path="/var/lib/kubelet/pods/54f291db-150a-4afb-98f4-a89003717a1b/volumes" Nov 01 06:48:10.814462 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:10.814355 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5"] Nov 01 06:48:10.815121 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:10.815077 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d307ac3_766f_40fd_a0da_f83c7fb4da7f.slice/crio-23fbc57b1da5362ee04b3d315ec1602369960d7f1de97d133fde5e7821882ce3 WatchSource:0}: Error finding container 23fbc57b1da5362ee04b3d315ec1602369960d7f1de97d133fde5e7821882ce3: Status 404 returned error can't find the container with id 23fbc57b1da5362ee04b3d315ec1602369960d7f1de97d133fde5e7821882ce3 Nov 01 06:48:11.767345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:11.767304 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" event={"ID":"6d307ac3-766f-40fd-a0da-f83c7fb4da7f","Type":"ContainerStarted","Data":"6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300"} Nov 01 06:48:11.767706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:11.767371 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:11.767706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:11.767391 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" event={"ID":"6d307ac3-766f-40fd-a0da-f83c7fb4da7f","Type":"ContainerStarted","Data":"23fbc57b1da5362ee04b3d315ec1602369960d7f1de97d133fde5e7821882ce3"} Nov 01 06:48:11.882496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:11.882447 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" podStartSLOduration=1.882434413 podStartE2EDuration="1.882434413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:11.879371824 +0000 UTC m=+2307.432161018" watchObservedRunningTime="2026-11-01 06:48:11.882434413 +0000 UTC m=+2307.435223606" Nov 01 06:48:14.709847 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:14.709805 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:17.741743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:17.741700 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:19.965706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:19.965675 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5"] Nov 01 06:48:19.968802 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:19.968762 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" podUID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:20.006238 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.006203 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp"] Nov 01 06:48:20.012990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.012966 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.079531 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.078883 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp"] Nov 01 06:48:20.113444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.113409 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.113604 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.113554 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.113645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.113628 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.113679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.113650 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smd6\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.214654 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.214600 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.214953 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.214912 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smd6\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.215110 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.215082 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.215265 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.215246 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.217667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.217256 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.218245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.218210 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.221088 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.221066 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.272292 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.272238 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smd6\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6\") pod \"payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.333716 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.333686 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.709127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.709105 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp"] Nov 01 06:48:20.710570 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:20.710547 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060d4f52_ef04_4d60_8e8e_20d996863586.slice/crio-d88c5ca7c04d1fb8956edb371a0915cf60501a41f127ea1e73df5eeb75dbe17c WatchSource:0}: Error finding container d88c5ca7c04d1fb8956edb371a0915cf60501a41f127ea1e73df5eeb75dbe17c: Status 404 returned error can't find the container with id d88c5ca7c04d1fb8956edb371a0915cf60501a41f127ea1e73df5eeb75dbe17c Nov 01 06:48:20.872940 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.872908 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" event={"ID":"060d4f52-ef04-4d60-8e8e-20d996863586","Type":"ContainerStarted","Data":"abc1456b1f4895e799933ac0cda9acd1aeb09f056a8cce2c5203ebba9e8fbca9"} Nov 01 06:48:20.873084 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.872965 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" event={"ID":"060d4f52-ef04-4d60-8e8e-20d996863586","Type":"ContainerStarted","Data":"d88c5ca7c04d1fb8956edb371a0915cf60501a41f127ea1e73df5eeb75dbe17c"} Nov 01 06:48:20.873084 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.872994 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:20.989472 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:20.989396 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" podStartSLOduration=1.989384715 podStartE2EDuration="1.989384715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:20.987588228 +0000 UTC m=+2316.540377421" watchObservedRunningTime="2026-11-01 06:48:20.989384715 +0000 UTC m=+2316.542173921" Nov 01 06:48:24.708117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:24.708074 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:24.965986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:24.965912 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" podUID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" containerName="payload-processing" containerID="cri-o://6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300" gracePeriod=25 Nov 01 06:48:25.233733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.233680 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:25.275411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.275387 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr\") pod \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " Nov 01 06:48:25.275541 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.275421 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume\") pod \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " Nov 01 06:48:25.275541 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.275450 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials\") pod \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " Nov 01 06:48:25.275667 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.275543 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume\") pod \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\" (UID: \"6d307ac3-766f-40fd-a0da-f83c7fb4da7f\") " Nov 01 06:48:25.275829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.275805 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume" pod "6d307ac3-766f-40fd-a0da-f83c7fb4da7f" (UID: "6d307ac3-766f-40fd-a0da-f83c7fb4da7f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:25.276068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.276043 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume" pod "6d307ac3-766f-40fd-a0da-f83c7fb4da7f" (UID: "6d307ac3-766f-40fd-a0da-f83c7fb4da7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:25.279323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.278045 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr" pod "6d307ac3-766f-40fd-a0da-f83c7fb4da7f" (UID: "6d307ac3-766f-40fd-a0da-f83c7fb4da7f"). InnerVolumeSpecName "kube-api-access-g8btr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:25.279432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.279407 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials" pod "6d307ac3-766f-40fd-a0da-f83c7fb4da7f" (UID: "6d307ac3-766f-40fd-a0da-f83c7fb4da7f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:25.376452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.376427 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8btr\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-kube-api-access-g8btr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:25.376578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.376455 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:25.376578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.376469 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:25.376578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.376480 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d307ac3-766f-40fd-a0da-f83c7fb4da7f-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:25.905057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.905020 4914 generic.go:356] "Generic (PLEG): container finished" podID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" containerID="6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300" exitCode=0 Nov 01 06:48:25.905450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.905067 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" Nov 01 06:48:25.905450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.905102 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" event={"ID":"6d307ac3-766f-40fd-a0da-f83c7fb4da7f","Type":"ContainerDied","Data":"6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300"} Nov 01 06:48:25.905450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.905149 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5" event={"ID":"6d307ac3-766f-40fd-a0da-f83c7fb4da7f","Type":"ContainerDied","Data":"23fbc57b1da5362ee04b3d315ec1602369960d7f1de97d133fde5e7821882ce3"} Nov 01 06:48:25.905450 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.905173 4914 scope.go:122] "RemoveContainer" containerID="6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300" Nov 01 06:48:25.915590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.915564 4914 scope.go:122] "RemoveContainer" containerID="6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300" Nov 01 06:48:25.915832 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:48:25.915810 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300\": container with ID starting with 6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300 not found: ID does not exist" containerID="6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300" Nov 01 06:48:25.915882 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:25.915845 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300"} err="failed to get container status \"6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300\": rpc error: code = NotFound desc = could not find container \"6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300\": container with ID starting with 6c254fa2cb01ea0ed56e6c732e53e382039bc43bbd546b1db5129796411f6300 not found: ID does not exist" Nov 01 06:48:26.068308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:26.068263 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5"] Nov 01 06:48:26.117556 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:26.117529 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-b788z2fl5"] Nov 01 06:48:26.616528 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:26.616500 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" path="/var/lib/kubelet/pods/6d307ac3-766f-40fd-a0da-f83c7fb4da7f/volumes" Nov 01 06:48:27.398859 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:27.398823 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb"] Nov 01 06:48:27.400106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:27.400078 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:30.085466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:30.085433 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p"] Nov 01 06:48:30.086777 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:30.086740 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:31.881290 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:31.881237 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:32.399368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.399323 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerName="payload-processing" containerID="cri-o://9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463" gracePeriod=25 Nov 01 06:48:32.677684 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.677657 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:32.761443 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761415 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume\") pod \"cc80f414-2593-4b63-8838-64d1f217d9bd\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " Nov 01 06:48:32.761585 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761485 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials\") pod \"cc80f414-2593-4b63-8838-64d1f217d9bd\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " Nov 01 06:48:32.761585 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761518 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume\") pod \"cc80f414-2593-4b63-8838-64d1f217d9bd\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " Nov 01 06:48:32.761699 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761640 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9\") pod \"cc80f414-2593-4b63-8838-64d1f217d9bd\" (UID: \"cc80f414-2593-4b63-8838-64d1f217d9bd\") " Nov 01 06:48:32.761824 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761797 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume" pod "cc80f414-2593-4b63-8838-64d1f217d9bd" (UID: "cc80f414-2593-4b63-8838-64d1f217d9bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:32.761924 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761907 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume" pod "cc80f414-2593-4b63-8838-64d1f217d9bd" (UID: "cc80f414-2593-4b63-8838-64d1f217d9bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:32.761994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.761977 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.763694 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.763666 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials" pod "cc80f414-2593-4b63-8838-64d1f217d9bd" (UID: "cc80f414-2593-4b63-8838-64d1f217d9bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:32.763858 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.763838 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9" pod "cc80f414-2593-4b63-8838-64d1f217d9bd" (UID: "cc80f414-2593-4b63-8838-64d1f217d9bd"). InnerVolumeSpecName "kube-api-access-4sfv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:32.781951 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.781928 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp"] Nov 01 06:48:32.783068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.783040 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:48:32.863609 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.863579 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.863609 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.863609 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc80f414-2593-4b63-8838-64d1f217d9bd-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.863821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.863623 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfv9\" (UniqueName: \"kubernetes.io/projected/cc80f414-2593-4b63-8838-64d1f217d9bd-kube-api-access-4sfv9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:32.952717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.952627 4914 generic.go:356] "Generic (PLEG): container finished" podID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerID="9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463" exitCode=0 Nov 01 06:48:32.952717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.952684 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" Nov 01 06:48:32.952717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.952693 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" event={"ID":"cc80f414-2593-4b63-8838-64d1f217d9bd","Type":"ContainerDied","Data":"9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463"} Nov 01 06:48:32.953317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.952759 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb" event={"ID":"cc80f414-2593-4b63-8838-64d1f217d9bd","Type":"ContainerDied","Data":"155b62dd3c8a0885d140ca0a3fb8a73093ffbb7f1bb83fbad86cc3bb910e2960"} Nov 01 06:48:32.953317 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.952781 4914 scope.go:122] "RemoveContainer" containerID="9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463" Nov 01 06:48:32.975403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.968480 4914 scope.go:122] "RemoveContainer" containerID="9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463" Nov 01 06:48:32.975403 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:48:32.968926 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463\": container with ID starting with 9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463 not found: ID does not exist" containerID="9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463" Nov 01 06:48:32.975403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:32.968975 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463"} err="failed to get container status \"9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463\": rpc error: code = NotFound desc = could not find container \"9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463\": container with ID starting with 9f80b85812ce5f6aad672f32703ffb23e512a47fe2c49041bfe7103cd36b9463 not found: ID does not exist" Nov 01 06:48:33.117674 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:33.117642 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb"] Nov 01 06:48:33.121148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:33.121122 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ff1dc/payload-processing-external-model-e2e-extmod-b-1ff1dc-c79f66chb"] Nov 01 06:48:34.616506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:34.616475 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" path="/var/lib/kubelet/pods/cc80f414-2593-4b63-8838-64d1f217d9bd/volumes" Nov 01 06:48:35.086187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.086069 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" containerID="cri-o://8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f" gracePeriod=25 Nov 01 06:48:35.356988 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.356965 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:35.389016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.388990 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials\") pod \"056002ce-fcc2-49a5-aea3-96592c8d4e58\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " Nov 01 06:48:35.389179 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.389074 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z\") pod \"056002ce-fcc2-49a5-aea3-96592c8d4e58\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " Nov 01 06:48:35.389249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.389191 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume\") pod \"056002ce-fcc2-49a5-aea3-96592c8d4e58\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " Nov 01 06:48:35.389249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.389222 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume\") pod \"056002ce-fcc2-49a5-aea3-96592c8d4e58\" (UID: \"056002ce-fcc2-49a5-aea3-96592c8d4e58\") " Nov 01 06:48:35.389671 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.389642 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume" pod "056002ce-fcc2-49a5-aea3-96592c8d4e58" (UID: "056002ce-fcc2-49a5-aea3-96592c8d4e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:35.389783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.389689 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume" pod "056002ce-fcc2-49a5-aea3-96592c8d4e58" (UID: "056002ce-fcc2-49a5-aea3-96592c8d4e58"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:35.391484 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.391461 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z" pod "056002ce-fcc2-49a5-aea3-96592c8d4e58" (UID: "056002ce-fcc2-49a5-aea3-96592c8d4e58"). InnerVolumeSpecName "kube-api-access-vqr9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:35.391647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.391477 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials" pod "056002ce-fcc2-49a5-aea3-96592c8d4e58" (UID: "056002ce-fcc2-49a5-aea3-96592c8d4e58"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:35.490736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.490704 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqr9z\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-kube-api-access-vqr9z\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.490736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.490732 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.490736 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.490743 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/056002ce-fcc2-49a5-aea3-96592c8d4e58-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.490952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.490751 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/056002ce-fcc2-49a5-aea3-96592c8d4e58-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:35.972580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.972546 4914 generic.go:356] "Generic (PLEG): container finished" podID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerID="8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f" exitCode=0 Nov 01 06:48:35.973005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.972584 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" Nov 01 06:48:35.973005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.972628 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" event={"ID":"056002ce-fcc2-49a5-aea3-96592c8d4e58","Type":"ContainerDied","Data":"8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f"} Nov 01 06:48:35.973005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.972685 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p" event={"ID":"056002ce-fcc2-49a5-aea3-96592c8d4e58","Type":"ContainerDied","Data":"a2bb551b0760dd98e9723358bde05a33dc65769efa816794cc05688c64641c21"} Nov 01 06:48:35.973005 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.972707 4914 scope.go:122] "RemoveContainer" containerID="8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f" Nov 01 06:48:35.984840 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.984822 4914 scope.go:122] "RemoveContainer" containerID="8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f" Nov 01 06:48:35.985122 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:48:35.985104 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f\": container with ID starting with 8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f not found: ID does not exist" containerID="8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f" Nov 01 06:48:35.985166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:35.985130 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f"} err="failed to get container status \"8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f\": rpc error: code = NotFound desc = could not find container \"8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f\": container with ID starting with 8a6dad38154462809e4caf38a3e29c572ed3b80bc9115a5d41caf58c2db5d16f not found: ID does not exist" Nov 01 06:48:36.135456 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:36.135425 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p"] Nov 01 06:48:36.184626 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:36.184599 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25719d/payload-processing-external-model-e2e-extmod-a-25719d-7cf6tcp8p"] Nov 01 06:48:36.625819 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:36.619701 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" path="/var/lib/kubelet/pods/056002ce-fcc2-49a5-aea3-96592c8d4e58/volumes" Nov 01 06:48:37.591438 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:48:37.591405 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:48:37.783099 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:37.783024 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" containerName="payload-processing" containerID="cri-o://abc1456b1f4895e799933ac0cda9acd1aeb09f056a8cce2c5203ebba9e8fbca9" gracePeriod=25 Nov 01 06:48:37.987306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:37.987253 4914 generic.go:356] "Generic (PLEG): container finished" podID="060d4f52-ef04-4d60-8e8e-20d996863586" containerID="abc1456b1f4895e799933ac0cda9acd1aeb09f056a8cce2c5203ebba9e8fbca9" exitCode=0 Nov 01 06:48:37.987463 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:37.987319 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" event={"ID":"060d4f52-ef04-4d60-8e8e-20d996863586","Type":"ContainerDied","Data":"abc1456b1f4895e799933ac0cda9acd1aeb09f056a8cce2c5203ebba9e8fbca9"} Nov 01 06:48:38.049918 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.049858 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:38.115655 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.115621 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume\") pod \"060d4f52-ef04-4d60-8e8e-20d996863586\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " Nov 01 06:48:38.115812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.115697 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume\") pod \"060d4f52-ef04-4d60-8e8e-20d996863586\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " Nov 01 06:48:38.115812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.115801 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials\") pod \"060d4f52-ef04-4d60-8e8e-20d996863586\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " Nov 01 06:48:38.115919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.115862 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6\") pod \"060d4f52-ef04-4d60-8e8e-20d996863586\" (UID: \"060d4f52-ef04-4d60-8e8e-20d996863586\") " Nov 01 06:48:38.116048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.116023 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume" pod "060d4f52-ef04-4d60-8e8e-20d996863586" (UID: "060d4f52-ef04-4d60-8e8e-20d996863586"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:38.116048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.116036 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume" pod "060d4f52-ef04-4d60-8e8e-20d996863586" (UID: "060d4f52-ef04-4d60-8e8e-20d996863586"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:48:38.116254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.116232 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:38.116254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.116251 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/060d4f52-ef04-4d60-8e8e-20d996863586-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:38.117965 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.117949 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials" pod "060d4f52-ef04-4d60-8e8e-20d996863586" (UID: "060d4f52-ef04-4d60-8e8e-20d996863586"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:38.118033 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.117997 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6" pod "060d4f52-ef04-4d60-8e8e-20d996863586" (UID: "060d4f52-ef04-4d60-8e8e-20d996863586"). InnerVolumeSpecName "kube-api-access-8smd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:48:38.217060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.217025 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:38.217060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.217054 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smd6\" (UniqueName: \"kubernetes.io/projected/060d4f52-ef04-4d60-8e8e-20d996863586-kube-api-access-8smd6\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:48:38.994939 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.994856 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" event={"ID":"060d4f52-ef04-4d60-8e8e-20d996863586","Type":"ContainerDied","Data":"d88c5ca7c04d1fb8956edb371a0915cf60501a41f127ea1e73df5eeb75dbe17c"} Nov 01 06:48:38.994939 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.994884 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp" Nov 01 06:48:38.994939 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:38.994924 4914 scope.go:122] "RemoveContainer" containerID="abc1456b1f4895e799933ac0cda9acd1aeb09f056a8cce2c5203ebba9e8fbca9" Nov 01 06:48:39.159021 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:39.158984 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp"] Nov 01 06:48:39.208676 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:39.208644 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dc6e48/payload-processing-external-model-e2e-extmod-b-dc6e48-ff7c4bwbp"] Nov 01 06:48:40.617699 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:40.617665 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" path="/var/lib/kubelet/pods/060d4f52-ef04-4d60-8e8e-20d996863586/volumes" Nov 01 06:48:46.413517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.413484 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm"] Nov 01 06:48:46.414075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414062 4914 cpu_manager.go:414] "containerMap: removing container" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" Nov 01 06:48:46.414125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414076 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" Nov 01 06:48:46.414125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414088 4914 cpu_manager.go:414] "containerMap: removing container" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerName="payload-processing" Nov 01 06:48:46.414125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414093 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerName="payload-processing" Nov 01 06:48:46.414125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414111 4914 cpu_manager.go:414] "containerMap: removing container" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" containerName="payload-processing" Nov 01 06:48:46.414125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414116 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" containerName="payload-processing" Nov 01 06:48:46.414356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414138 4914 cpu_manager.go:414] "containerMap: removing container" podUID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" containerName="payload-processing" Nov 01 06:48:46.414356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414142 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" containerName="payload-processing" Nov 01 06:48:46.414356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414232 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="056002ce-fcc2-49a5-aea3-96592c8d4e58" containerName="payload-processing" Nov 01 06:48:46.414356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414240 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc80f414-2593-4b63-8838-64d1f217d9bd" containerName="payload-processing" Nov 01 06:48:46.414356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414249 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="060d4f52-ef04-4d60-8e8e-20d996863586" containerName="payload-processing" Nov 01 06:48:46.414356 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.414258 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d307ac3-766f-40fd-a0da-f83c7fb4da7f" containerName="payload-processing" Nov 01 06:48:46.419050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.419035 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.522417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522377 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-95e911575933\"" Nov 01 06:48:46.522417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522377 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 06:48:46.522417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522385 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 06:48:46.522662 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522448 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1b7c4c0142f1\"" Nov 01 06:48:46.522662 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522408 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 06:48:46.522662 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522424 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 06:48:46.522662 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.522476 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tng78\"" Nov 01 06:48:46.525176 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.525156 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm"] Nov 01 06:48:46.531669 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.531647 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.531786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.531686 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nql\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.531848 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.531822 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.531905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.531882 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.633270 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.633233 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.633270 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.633304 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.633492 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.633378 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.633492 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.633408 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nql\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.634086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.634058 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.634211 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.634125 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.635840 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.635817 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.689812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.689755 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nql\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql\") pod \"payload-processing-external-model-7446cdb47f-xqntm\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:46.732776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:46.732753 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:47.150188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:47.150162 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm"] Nov 01 06:48:47.152011 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:47.151978 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ac7783_7543_411a_80af_8a0b236d468f.slice/crio-aa91bebec2b071b4d3d548dabbb3d3847d786a6a800ca5fa92c3f7798d9bd111 WatchSource:0}: Error finding container aa91bebec2b071b4d3d548dabbb3d3847d786a6a800ca5fa92c3f7798d9bd111: Status 404 returned error can't find the container with id aa91bebec2b071b4d3d548dabbb3d3847d786a6a800ca5fa92c3f7798d9bd111 Nov 01 06:48:48.052009 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:48.051975 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" event={"ID":"49ac7783-7543-411a-80af-8a0b236d468f","Type":"ContainerStarted","Data":"1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04"} Nov 01 06:48:48.052413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:48.052029 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" event={"ID":"49ac7783-7543-411a-80af-8a0b236d468f","Type":"ContainerStarted","Data":"aa91bebec2b071b4d3d548dabbb3d3847d786a6a800ca5fa92c3f7798d9bd111"} Nov 01 06:48:48.052413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:48.052087 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:48:48.166170 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:48.166118 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" podStartSLOduration=2.166104603 podStartE2EDuration="2.166104603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:48.164266842 +0000 UTC m=+2343.717056034" watchObservedRunningTime="2026-11-01 06:48:48.166104603 +0000 UTC m=+2343.718893795" Nov 01 06:48:51.374850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.374813 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2"] Nov 01 06:48:51.379509 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.379487 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.431776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.431739 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2"] Nov 01 06:48:51.457675 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.457627 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56p6h\" (UniqueName: \"kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h\") pod \"payload-processing-e2e-extmod-b-581487-f88795567-8kgw2\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.457832 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.457782 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-581487-f88795567-8kgw2\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.478065 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.478041 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-581487\"" Nov 01 06:48:51.478201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.478092 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-581487-dockercfg-22gf8\"" Nov 01 06:48:51.558494 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.558460 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56p6h\" (UniqueName: \"kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h\") pod \"payload-processing-e2e-extmod-b-581487-f88795567-8kgw2\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.558652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.558561 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-581487-f88795567-8kgw2\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.559249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.559227 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-581487-f88795567-8kgw2\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.614860 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.614827 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56p6h\" (UniqueName: \"kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h\") pod \"payload-processing-e2e-extmod-b-581487-f88795567-8kgw2\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:51.696996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:51.696921 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:52.066623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:52.066597 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2"] Nov 01 06:48:52.068115 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:52.068093 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5940d4_266f_484e_8f49_1edbb557108a.slice/crio-6f3de7516535f18d8d68a5d6ac9b322c8fb228383f725ad049ae36cf6b4a5dde WatchSource:0}: Error finding container 6f3de7516535f18d8d68a5d6ac9b322c8fb228383f725ad049ae36cf6b4a5dde: Status 404 returned error can't find the container with id 6f3de7516535f18d8d68a5d6ac9b322c8fb228383f725ad049ae36cf6b4a5dde Nov 01 06:48:52.099975 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:52.099941 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" event={"ID":"1e5940d4-266f-484e-8f49-1edbb557108a","Type":"ContainerStarted","Data":"6f3de7516535f18d8d68a5d6ac9b322c8fb228383f725ad049ae36cf6b4a5dde"} Nov 01 06:48:53.065245 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.065209 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9"] Nov 01 06:48:53.069771 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.069754 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.106938 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.106909 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" event={"ID":"1e5940d4-266f-484e-8f49-1edbb557108a","Type":"ContainerStarted","Data":"03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19"} Nov 01 06:48:53.107077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.106970 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:48:53.123911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.123891 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6883279a9766\"" Nov 01 06:48:53.176596 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.176552 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9"] Nov 01 06:48:53.183880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.183851 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgglp\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.184031 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.183896 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.184125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.184102 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.184226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.184210 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.285561 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.285522 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.285750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.285644 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgglp\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.285750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.285675 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.285856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.285749 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.286259 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.286232 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.286536 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.286516 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.288446 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.288423 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.297202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.297157 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" podStartSLOduration=2.297143053 podStartE2EDuration="2.297143053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:53.295006663 +0000 UTC m=+2348.847795857" watchObservedRunningTime="2026-11-01 06:48:53.297143053 +0000 UTC m=+2348.849932246" Nov 01 06:48:53.342487 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.342417 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgglp\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp\") pod \"payload-processing-external-model-595cb556d4-tfnm9\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.383559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.383510 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:53.788614 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:53.788582 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9"] Nov 01 06:48:53.789375 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:48:53.789344 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a65a4a_3a31_42c3_89fe_88518f2ac9ac.slice/crio-e323144aa1de48fb77c9edc66971a57ee2662808c2f4bb70a611b37b9e56984a WatchSource:0}: Error finding container e323144aa1de48fb77c9edc66971a57ee2662808c2f4bb70a611b37b9e56984a: Status 404 returned error can't find the container with id e323144aa1de48fb77c9edc66971a57ee2662808c2f4bb70a611b37b9e56984a Nov 01 06:48:54.114095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:54.114056 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" event={"ID":"45a65a4a-3a31-42c3-89fe-88518f2ac9ac","Type":"ContainerStarted","Data":"417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c"} Nov 01 06:48:54.114592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:54.114114 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" event={"ID":"45a65a4a-3a31-42c3-89fe-88518f2ac9ac","Type":"ContainerStarted","Data":"e323144aa1de48fb77c9edc66971a57ee2662808c2f4bb70a611b37b9e56984a"} Nov 01 06:48:54.114592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:54.114140 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:48:54.228182 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:54.228131 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" podStartSLOduration=1.228118196 podStartE2EDuration="1.228118196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:48:54.225477004 +0000 UTC m=+2349.778266197" watchObservedRunningTime="2026-11-01 06:48:54.228118196 +0000 UTC m=+2349.780907388" Nov 01 06:48:59.060626 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:48:59.060593 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:49:04.116896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:04.116856 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 06:49:05.123675 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:05.123649 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:49:05.288254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:05.288224 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm"] Nov 01 06:49:07.597378 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:07.597343 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:49:10.288839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.288793 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" podUID="49ac7783-7543-411a-80af-8a0b236d468f" containerName="payload-processing" containerID="cri-o://1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04" gracePeriod=25 Nov 01 06:49:10.543669 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.543601 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:49:10.626542 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.626511 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials\") pod \"49ac7783-7543-411a-80af-8a0b236d468f\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " Nov 01 06:49:10.626693 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.626549 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql\") pod \"49ac7783-7543-411a-80af-8a0b236d468f\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " Nov 01 06:49:10.626693 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.626684 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume\") pod \"49ac7783-7543-411a-80af-8a0b236d468f\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " Nov 01 06:49:10.626842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.626828 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume\") pod \"49ac7783-7543-411a-80af-8a0b236d468f\" (UID: \"49ac7783-7543-411a-80af-8a0b236d468f\") " Nov 01 06:49:10.627076 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.627048 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume" pod "49ac7783-7543-411a-80af-8a0b236d468f" (UID: "49ac7783-7543-411a-80af-8a0b236d468f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:10.627185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.627143 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume" pod "49ac7783-7543-411a-80af-8a0b236d468f" (UID: "49ac7783-7543-411a-80af-8a0b236d468f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:10.627329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.627313 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:10.627381 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.627332 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49ac7783-7543-411a-80af-8a0b236d468f-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:10.628704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.628683 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials" pod "49ac7783-7543-411a-80af-8a0b236d468f" (UID: "49ac7783-7543-411a-80af-8a0b236d468f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:10.628810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.628761 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql" pod "49ac7783-7543-411a-80af-8a0b236d468f" (UID: "49ac7783-7543-411a-80af-8a0b236d468f"). InnerVolumeSpecName "kube-api-access-k6nql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:10.728059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.728032 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:10.728059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:10.728057 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6nql\" (UniqueName: \"kubernetes.io/projected/49ac7783-7543-411a-80af-8a0b236d468f-kube-api-access-k6nql\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:11.235002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.234968 4914 generic.go:356] "Generic (PLEG): container finished" podID="49ac7783-7543-411a-80af-8a0b236d468f" containerID="1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04" exitCode=0 Nov 01 06:49:11.235172 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.235013 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" Nov 01 06:49:11.235172 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.235056 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" event={"ID":"49ac7783-7543-411a-80af-8a0b236d468f","Type":"ContainerDied","Data":"1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04"} Nov 01 06:49:11.235172 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.235111 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm" event={"ID":"49ac7783-7543-411a-80af-8a0b236d468f","Type":"ContainerDied","Data":"aa91bebec2b071b4d3d548dabbb3d3847d786a6a800ca5fa92c3f7798d9bd111"} Nov 01 06:49:11.235172 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.235133 4914 scope.go:122] "RemoveContainer" containerID="1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04" Nov 01 06:49:11.248089 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.248072 4914 scope.go:122] "RemoveContainer" containerID="1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04" Nov 01 06:49:11.248353 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:11.248331 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04\": container with ID starting with 1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04 not found: ID does not exist" containerID="1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04" Nov 01 06:49:11.248420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.248357 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04"} err="failed to get container status \"1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04\": rpc error: code = NotFound desc = could not find container \"1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04\": container with ID starting with 1c0872e22947361dee453271c4bdbb9a104731c10d160f2a1da2bc9475758d04 not found: ID does not exist" Nov 01 06:49:11.398179 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.398151 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm"] Nov 01 06:49:11.405628 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:11.405607 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7446cdb47f-xqntm"] Nov 01 06:49:12.616866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:12.616836 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ac7783-7543-411a-80af-8a0b236d468f" path="/var/lib/kubelet/pods/49ac7783-7543-411a-80af-8a0b236d468f/volumes" Nov 01 06:49:24.656538 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.656483 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj"] Nov 01 06:49:24.657180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.657164 4914 cpu_manager.go:414] "containerMap: removing container" podUID="49ac7783-7543-411a-80af-8a0b236d468f" containerName="payload-processing" Nov 01 06:49:24.657180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.657181 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ac7783-7543-411a-80af-8a0b236d468f" containerName="payload-processing" Nov 01 06:49:24.657315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.657304 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ac7783-7543-411a-80af-8a0b236d468f" containerName="payload-processing" Nov 01 06:49:24.661500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.661485 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.767369 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.767338 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj"] Nov 01 06:49:24.785908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.785876 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.786057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.786000 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.786117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.786064 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.786117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.786085 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66c7\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.887479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.887443 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.887479 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.887483 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66c7\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.887682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.887550 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.887682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.887599 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.888157 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.888133 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.888256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.888198 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.890269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.890244 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.943329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.943248 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66c7\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7\") pod \"payload-processing-external-model-787d5c596d-gsfjj\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:24.975316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:24.975267 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:25.350014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:25.349986 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj"] Nov 01 06:49:25.350727 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:49:25.350704 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e5c998_eeae_4c11_b0b8_a2ba947ee036.slice/crio-ee76844ad1a41c891f8c3918f2286341d4c5f8df32f5e657372b03dcc35aa995 WatchSource:0}: Error finding container ee76844ad1a41c891f8c3918f2286341d4c5f8df32f5e657372b03dcc35aa995: Status 404 returned error can't find the container with id ee76844ad1a41c891f8c3918f2286341d4c5f8df32f5e657372b03dcc35aa995 Nov 01 06:49:25.378524 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:25.378500 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" event={"ID":"49e5c998-eeae-4c11-b0b8-a2ba947ee036","Type":"ContainerStarted","Data":"ee76844ad1a41c891f8c3918f2286341d4c5f8df32f5e657372b03dcc35aa995"} Nov 01 06:49:26.023139 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:26.023057 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476"] Nov 01 06:49:26.126692 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:26.126001 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt"] Nov 01 06:49:26.386443 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:26.386408 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" event={"ID":"49e5c998-eeae-4c11-b0b8-a2ba947ee036","Type":"ContainerStarted","Data":"1221bb99c05a49c2a73aa3455bc13b19627899e733e4a27b6bc48cd6f54c67f7"} Nov 01 06:49:26.386636 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:26.386507 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:49:26.518031 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:26.517982 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podStartSLOduration=2.517968055 podStartE2EDuration="2.517968055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:49:26.517054412 +0000 UTC m=+2382.069843605" watchObservedRunningTime="2026-11-01 06:49:26.517968055 +0000 UTC m=+2382.070757247" Nov 01 06:49:31.025403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.025345 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" podUID="3b36695f-0e08-451d-9a9b-36e811b788db" containerName="payload-processing" containerID="cri-o://cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85" gracePeriod=25 Nov 01 06:49:31.126440 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.126382 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" podUID="69566da2-0890-4d50-9ecc-61f4b24d5760" containerName="payload-pre-processing" containerID="cri-o://973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0" gracePeriod=25 Nov 01 06:49:31.333262 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.333216 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:49:31.434627 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.434598 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:49:31.453996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.453960 4914 generic.go:356] "Generic (PLEG): container finished" podID="3b36695f-0e08-451d-9a9b-36e811b788db" containerID="cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85" exitCode=0 Nov 01 06:49:31.454177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.454029 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" Nov 01 06:49:31.454177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.454056 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" event={"ID":"3b36695f-0e08-451d-9a9b-36e811b788db","Type":"ContainerDied","Data":"cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85"} Nov 01 06:49:31.454177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.454114 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476" event={"ID":"3b36695f-0e08-451d-9a9b-36e811b788db","Type":"ContainerDied","Data":"20a6981c8f953ca7af2cc13b63810b02875b5e6a9848bba1fde30e3c46a77bca"} Nov 01 06:49:31.454177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.454142 4914 scope.go:122] "RemoveContainer" containerID="cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85" Nov 01 06:49:31.455876 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.455857 4914 generic.go:356] "Generic (PLEG): container finished" podID="69566da2-0890-4d50-9ecc-61f4b24d5760" containerID="973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0" exitCode=0 Nov 01 06:49:31.455976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.455900 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" event={"ID":"69566da2-0890-4d50-9ecc-61f4b24d5760","Type":"ContainerDied","Data":"973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0"} Nov 01 06:49:31.455976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.455949 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" event={"ID":"69566da2-0890-4d50-9ecc-61f4b24d5760","Type":"ContainerDied","Data":"f455925005397553cbfa7c5f9feb269b993f5f53f26d2c649075320d00d087f3"} Nov 01 06:49:31.456125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.456106 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt" Nov 01 06:49:31.464300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.464248 4914 scope.go:122] "RemoveContainer" containerID="cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85" Nov 01 06:49:31.464626 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:31.464597 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85\": container with ID starting with cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85 not found: ID does not exist" containerID="cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85" Nov 01 06:49:31.464717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.464632 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85"} err="failed to get container status \"cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85\": rpc error: code = NotFound desc = could not find container \"cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85\": container with ID starting with cb8cc9f9753799646919666c064935bb7396942c30cdc01ca3c54f479cbdfd85 not found: ID does not exist" Nov 01 06:49:31.464717 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.464649 4914 scope.go:122] "RemoveContainer" containerID="973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0" Nov 01 06:49:31.473924 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.473908 4914 scope.go:122] "RemoveContainer" containerID="973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0" Nov 01 06:49:31.474180 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:31.474152 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0\": container with ID starting with 973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0 not found: ID does not exist" containerID="973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0" Nov 01 06:49:31.474235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.474186 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0"} err="failed to get container status \"973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0\": rpc error: code = NotFound desc = could not find container \"973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0\": container with ID starting with 973ca1971cbaa7b595e471f35d72ac388a0632344720857138831afeb71b0de0 not found: ID does not exist" Nov 01 06:49:31.495776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.495738 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume\") pod \"3b36695f-0e08-451d-9a9b-36e811b788db\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " Nov 01 06:49:31.496076 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.496032 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv\" (UniqueName: \"kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv\") pod \"3b36695f-0e08-451d-9a9b-36e811b788db\" (UID: \"3b36695f-0e08-451d-9a9b-36e811b788db\") " Nov 01 06:49:31.496175 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.496144 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume" pod "3b36695f-0e08-451d-9a9b-36e811b788db" (UID: "3b36695f-0e08-451d-9a9b-36e811b788db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:31.496419 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.496403 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b36695f-0e08-451d-9a9b-36e811b788db-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:31.498698 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.498677 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv" pod "3b36695f-0e08-451d-9a9b-36e811b788db" (UID: "3b36695f-0e08-451d-9a9b-36e811b788db"). InnerVolumeSpecName "kube-api-access-2q8fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:31.596803 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.596771 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc\" (UniqueName: \"kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc\") pod \"69566da2-0890-4d50-9ecc-61f4b24d5760\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " Nov 01 06:49:31.596970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.596933 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume\") pod \"69566da2-0890-4d50-9ecc-61f4b24d5760\" (UID: \"69566da2-0890-4d50-9ecc-61f4b24d5760\") " Nov 01 06:49:31.597354 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.597327 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume" pod "69566da2-0890-4d50-9ecc-61f4b24d5760" (UID: "69566da2-0890-4d50-9ecc-61f4b24d5760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:31.597456 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.597396 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8fv\" (UniqueName: \"kubernetes.io/projected/3b36695f-0e08-451d-9a9b-36e811b788db-kube-api-access-2q8fv\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:31.599067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.599047 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc" pod "69566da2-0890-4d50-9ecc-61f4b24d5760" (UID: "69566da2-0890-4d50-9ecc-61f4b24d5760"). InnerVolumeSpecName "kube-api-access-8n9kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:31.698406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.698378 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9kc\" (UniqueName: \"kubernetes.io/projected/69566da2-0890-4d50-9ecc-61f4b24d5760-kube-api-access-8n9kc\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:31.698406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.698407 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69566da2-0890-4d50-9ecc-61f4b24d5760-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:31.875925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.875860 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476"] Nov 01 06:49:31.879584 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.879562 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47bc8f48-65dd45f78c-8n476"] Nov 01 06:49:31.987365 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.987331 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt"] Nov 01 06:49:31.992788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:31.992764 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47bc8f48-789f69b859-sk2rt"] Nov 01 06:49:32.066650 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:32.066628 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c"] Nov 01 06:49:32.179765 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:32.179669 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv"] Nov 01 06:49:32.632661 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:32.632629 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b36695f-0e08-451d-9a9b-36e811b788db" path="/var/lib/kubelet/pods/3b36695f-0e08-451d-9a9b-36e811b788db/volumes" Nov 01 06:49:32.633036 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:32.633021 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69566da2-0890-4d50-9ecc-61f4b24d5760" path="/var/lib/kubelet/pods/69566da2-0890-4d50-9ecc-61f4b24d5760/volumes" Nov 01 06:49:37.067622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.067582 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" podUID="85082994-b04d-4774-bb1e-42119ba6b298" containerName="payload-processing" containerID="cri-o://e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338" gracePeriod=25 Nov 01 06:49:37.180548 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.180517 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" podUID="103a6ac1-a23b-4cb9-8db9-124d33683828" containerName="payload-pre-processing" containerID="cri-o://4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1" gracePeriod=25 Nov 01 06:49:37.340906 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.340850 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:49:37.430058 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.430000 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:49:37.440106 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.440082 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:49:37.490350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.490325 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99\" (UniqueName: \"kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99\") pod \"85082994-b04d-4774-bb1e-42119ba6b298\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " Nov 01 06:49:37.490483 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.490363 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume\") pod \"85082994-b04d-4774-bb1e-42119ba6b298\" (UID: \"85082994-b04d-4774-bb1e-42119ba6b298\") " Nov 01 06:49:37.490826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.490798 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume" pod "85082994-b04d-4774-bb1e-42119ba6b298" (UID: "85082994-b04d-4774-bb1e-42119ba6b298"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:37.492723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.492702 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99" pod "85082994-b04d-4774-bb1e-42119ba6b298" (UID: "85082994-b04d-4774-bb1e-42119ba6b298"). InnerVolumeSpecName "kube-api-access-lkb99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:37.502223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.502200 4914 generic.go:356] "Generic (PLEG): container finished" podID="85082994-b04d-4774-bb1e-42119ba6b298" containerID="e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338" exitCode=0 Nov 01 06:49:37.502316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.502241 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" Nov 01 06:49:37.502316 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.502299 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" event={"ID":"85082994-b04d-4774-bb1e-42119ba6b298","Type":"ContainerDied","Data":"e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338"} Nov 01 06:49:37.502401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.502354 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c" event={"ID":"85082994-b04d-4774-bb1e-42119ba6b298","Type":"ContainerDied","Data":"b9333c134eb3220e719ca454d92735d80fdf2a7063f28c0664967ac81bc63ad1"} Nov 01 06:49:37.502442 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.502380 4914 scope.go:122] "RemoveContainer" containerID="e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338" Nov 01 06:49:37.503979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.503960 4914 generic.go:356] "Generic (PLEG): container finished" podID="103a6ac1-a23b-4cb9-8db9-124d33683828" containerID="4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1" exitCode=0 Nov 01 06:49:37.504068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.504025 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" event={"ID":"103a6ac1-a23b-4cb9-8db9-124d33683828","Type":"ContainerDied","Data":"4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1"} Nov 01 06:49:37.504068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.504056 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" event={"ID":"103a6ac1-a23b-4cb9-8db9-124d33683828","Type":"ContainerDied","Data":"84d24d9ff284f1bf7052f0f614a3ff4451b05c011aaca6f061fff56932393b69"} Nov 01 06:49:37.504149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.504107 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv" Nov 01 06:49:37.511914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.511898 4914 scope.go:122] "RemoveContainer" containerID="e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338" Nov 01 06:49:37.512201 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:37.512185 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338\": container with ID starting with e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338 not found: ID does not exist" containerID="e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338" Nov 01 06:49:37.512243 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.512209 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338"} err="failed to get container status \"e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338\": rpc error: code = NotFound desc = could not find container \"e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338\": container with ID starting with e01c6afee51aa9b0ed565d9592a4dc72c67c7b619af9fe5c1448aed335d91338 not found: ID does not exist" Nov 01 06:49:37.512243 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.512224 4914 scope.go:122] "RemoveContainer" containerID="4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1" Nov 01 06:49:37.521430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.521412 4914 scope.go:122] "RemoveContainer" containerID="4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1" Nov 01 06:49:37.521684 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:37.521667 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1\": container with ID starting with 4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1 not found: ID does not exist" containerID="4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1" Nov 01 06:49:37.521744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.521690 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1"} err="failed to get container status \"4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1\": rpc error: code = NotFound desc = could not find container \"4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1\": container with ID starting with 4fa33169e240ce360b4e4aecd2a6c16d689c845ecbd68cc72a74993ed3d34ff1 not found: ID does not exist" Nov 01 06:49:37.591368 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.591295 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w\" (UniqueName: \"kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w\") pod \"103a6ac1-a23b-4cb9-8db9-124d33683828\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " Nov 01 06:49:37.591504 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.591380 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume\") pod \"103a6ac1-a23b-4cb9-8db9-124d33683828\" (UID: \"103a6ac1-a23b-4cb9-8db9-124d33683828\") " Nov 01 06:49:37.591732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.591718 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkb99\" (UniqueName: \"kubernetes.io/projected/85082994-b04d-4774-bb1e-42119ba6b298-kube-api-access-lkb99\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:37.591732 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.591733 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85082994-b04d-4774-bb1e-42119ba6b298-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:37.591853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.591760 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume" pod "103a6ac1-a23b-4cb9-8db9-124d33683828" (UID: "103a6ac1-a23b-4cb9-8db9-124d33683828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:37.593358 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.593331 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w" pod "103a6ac1-a23b-4cb9-8db9-124d33683828" (UID: "103a6ac1-a23b-4cb9-8db9-124d33683828"). InnerVolumeSpecName "kube-api-access-5cw2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:37.597560 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:37.597529 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:49:37.675038 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.675014 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c"] Nov 01 06:49:37.677775 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.677752 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8a9760-998649d7c-lz57c"] Nov 01 06:49:37.692559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.692532 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cw2w\" (UniqueName: \"kubernetes.io/projected/103a6ac1-a23b-4cb9-8db9-124d33683828-kube-api-access-5cw2w\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:37.692559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.692558 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103a6ac1-a23b-4cb9-8db9-124d33683828-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:37.922958 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.922933 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv"] Nov 01 06:49:37.925477 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:37.925458 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8a9760-588ff9ff9-8cssv"] Nov 01 06:49:38.616443 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:38.616413 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103a6ac1-a23b-4cb9-8db9-124d33683828" path="/var/lib/kubelet/pods/103a6ac1-a23b-4cb9-8db9-124d33683828/volumes" Nov 01 06:49:38.616818 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:38.616785 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85082994-b04d-4774-bb1e-42119ba6b298" path="/var/lib/kubelet/pods/85082994-b04d-4774-bb1e-42119ba6b298/volumes" Nov 01 06:49:44.893075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:44.893049 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:49:44.900652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:44.900629 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:49:47.428898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:47.428862 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:49:53.459412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.459380 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p"] Nov 01 06:49:53.459957 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.459944 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3b36695f-0e08-451d-9a9b-36e811b788db" containerName="payload-processing" Nov 01 06:49:53.459957 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.459957 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b36695f-0e08-451d-9a9b-36e811b788db" containerName="payload-processing" Nov 01 06:49:53.460053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.459978 4914 cpu_manager.go:414] "containerMap: removing container" podUID="69566da2-0890-4d50-9ecc-61f4b24d5760" containerName="payload-pre-processing" Nov 01 06:49:53.460053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.459983 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69566da2-0890-4d50-9ecc-61f4b24d5760" containerName="payload-pre-processing" Nov 01 06:49:53.460053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.459996 4914 cpu_manager.go:414] "containerMap: removing container" podUID="103a6ac1-a23b-4cb9-8db9-124d33683828" containerName="payload-pre-processing" Nov 01 06:49:53.460053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460000 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103a6ac1-a23b-4cb9-8db9-124d33683828" containerName="payload-pre-processing" Nov 01 06:49:53.460053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460010 4914 cpu_manager.go:414] "containerMap: removing container" podUID="85082994-b04d-4774-bb1e-42119ba6b298" containerName="payload-processing" Nov 01 06:49:53.460053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460015 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85082994-b04d-4774-bb1e-42119ba6b298" containerName="payload-processing" Nov 01 06:49:53.460246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460111 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="85082994-b04d-4774-bb1e-42119ba6b298" containerName="payload-processing" Nov 01 06:49:53.460246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460125 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b36695f-0e08-451d-9a9b-36e811b788db" containerName="payload-processing" Nov 01 06:49:53.460246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460133 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="69566da2-0890-4d50-9ecc-61f4b24d5760" containerName="payload-pre-processing" Nov 01 06:49:53.460246 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.460140 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="103a6ac1-a23b-4cb9-8db9-124d33683828" containerName="payload-pre-processing" Nov 01 06:49:53.464845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.464824 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.547381 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.547347 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.547571 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.547461 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.547643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.547571 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.547643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.547636 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkn7f\" (UniqueName: \"kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.566409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.566375 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-581487-serving-cert\"" Nov 01 06:49:53.569523 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.569497 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p"] Nov 01 06:49:53.649230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.649195 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.649400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.649302 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.649400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.649332 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn7f\" (UniqueName: \"kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.649400 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:53.649355 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-serving-cert: secret "maas-api-e2e-extmod-b-581487-serving-cert" not found Nov 01 06:49:53.649528 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:53.649409 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls podName:8d2a08b9-4216-4aa9-bb72-3976810f685a nodeName:}" failed. No retries permitted until 2026-11-01 06:49:54.149393503 +0000 UTC m=+2409.702182678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls") pod "maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" (UID: "8d2a08b9-4216-4aa9-bb72-3976810f685a") : secret "maas-api-e2e-extmod-b-581487-serving-cert" not found Nov 01 06:49:53.649528 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.649360 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.650142 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.650069 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.651938 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.651914 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:53.732533 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:53.732433 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn7f\" (UniqueName: \"kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:54.168911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:54.168879 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:54.171466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:54.171438 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:54.378111 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:54.378081 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:54.751338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:54.751310 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p"] Nov 01 06:49:54.752768 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:49:54.752733 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2a08b9_4216_4aa9_bb72_3976810f685a.slice/crio-999731002e7a05098bd8a33a054785173f0b4c6321da3c57c568836b14056c72 WatchSource:0}: Error finding container 999731002e7a05098bd8a33a054785173f0b4c6321da3c57c568836b14056c72: Status 404 returned error can't find the container with id 999731002e7a05098bd8a33a054785173f0b4c6321da3c57c568836b14056c72 Nov 01 06:49:55.614000 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:55.613969 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" event={"ID":"8d2a08b9-4216-4aa9-bb72-3976810f685a","Type":"ContainerStarted","Data":"999731002e7a05098bd8a33a054785173f0b4c6321da3c57c568836b14056c72"} Nov 01 06:49:56.622045 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:56.622009 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" event={"ID":"8d2a08b9-4216-4aa9-bb72-3976810f685a","Type":"ContainerStarted","Data":"e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650"} Nov 01 06:49:56.622534 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:56.622098 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:49:56.736199 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:56.736140 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" podStartSLOduration=2.759803024 podStartE2EDuration="3.736126132s" totalImagesPullingTime="976.323108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:49:56.733933369 +0000 UTC m=+2412.286722553" watchObservedRunningTime="2026-11-01 06:49:56.736126132 +0000 UTC m=+2412.288915324" Nov 01 06:49:57.428864 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:57.428832 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:49:57.816699 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:57.816615 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh"] Nov 01 06:49:57.817160 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:57.816787 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" podUID="3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" containerName="maas-api" containerID="cri-o://df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda" gracePeriod=30 Nov 01 06:49:58.109726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.109701 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:49:58.241299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.240461 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs\") pod \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " Nov 01 06:49:58.241299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.240668 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58\" (UniqueName: \"kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58\") pod \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " Nov 01 06:49:58.241299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.240728 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\") pod \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " Nov 01 06:49:58.241299 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.240814 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca\") pod \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\" (UID: \"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02\") " Nov 01 06:49:58.241652 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.241548 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca" pod "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" (UID: "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:49:58.245447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.245402 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls" pod "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" (UID: "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:58.249811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.249778 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs" pod "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" (UID: "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:49:58.251654 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.251631 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58" pod "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" (UID: "3a01b668-5a0f-4a6c-9c67-8e6161dcbc02"). InnerVolumeSpecName "kube-api-access-kmq58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:49:58.361529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.352220 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:58.361529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.352260 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:58.361529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.352299 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmq58\" (UniqueName: \"kubernetes.io/projected/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-kube-api-access-kmq58\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:58.361529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.352315 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:49:58.635466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.635421 4914 generic.go:356] "Generic (PLEG): container finished" podID="3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" containerID="df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda" exitCode=0 Nov 01 06:49:58.635466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.635463 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" Nov 01 06:49:58.635670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.635497 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" event={"ID":"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02","Type":"ContainerDied","Data":"df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda"} Nov 01 06:49:58.635670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.635542 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh" event={"ID":"3a01b668-5a0f-4a6c-9c67-8e6161dcbc02","Type":"ContainerDied","Data":"44c81b33079f1bc95cfab4b161718b7d7d79cac19b9a168d319776fb7d142680"} Nov 01 06:49:58.635670 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.635563 4914 scope.go:122] "RemoveContainer" containerID="df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda" Nov 01 06:49:58.645068 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.645052 4914 scope.go:122] "RemoveContainer" containerID="df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda" Nov 01 06:49:58.645330 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:49:58.645309 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda\": container with ID starting with df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda not found: ID does not exist" containerID="df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda" Nov 01 06:49:58.645396 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.645338 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda"} err="failed to get container status \"df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda\": rpc error: code = NotFound desc = could not find container \"df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda\": container with ID starting with df5737d8e60c09b1c4bfb80e448f3ed3c92c6e806273850ad1ffcc1bc7b7aeda not found: ID does not exist" Nov 01 06:49:58.821445 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.821406 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh"] Nov 01 06:49:58.826205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:49:58.826175 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47bc8f48-555cc6df7c-d62kh"] Nov 01 06:50:00.616557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:00.616522 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" path="/var/lib/kubelet/pods/3a01b668-5a0f-4a6c-9c67-8e6161dcbc02/volumes" Nov 01 06:50:02.633375 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:02.633350 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 06:50:07.429358 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:07.429318 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:50:07.592716 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:07.592683 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:50:08.562909 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.562875 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj"] Nov 01 06:50:08.564053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.564023 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:50:08.587930 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.587907 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2"] Nov 01 06:50:08.588521 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.588478 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" containerName="maas-api" Nov 01 06:50:08.588521 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.588493 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" containerName="maas-api" Nov 01 06:50:08.588656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.588606 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a01b668-5a0f-4a6c-9c67-8e6161dcbc02" containerName="maas-api" Nov 01 06:50:08.593402 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.593387 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.687862 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.687830 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2"] Nov 01 06:50:08.693418 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.693384 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.693588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.693507 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.693669 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.693601 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.693770 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.693752 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr47\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.794446 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.794412 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.794642 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.794486 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.794642 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.794563 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.794766 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.794645 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr47\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.795353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.795320 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.795465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.795378 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.797097 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.797075 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.850700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.850680 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr47\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47\") pod \"payload-processing-external-model-7b968cbf54-hrgz2\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:08.906292 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:08.906248 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:09.278269 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:09.278239 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2"] Nov 01 06:50:09.279755 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:50:09.279730 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e92597_f951_4c28_b8df_0e4155eae502.slice/crio-537fd2da3c56aa22ff6a4c5fe7429a9e408971101160a6bf36d551d69c636242 WatchSource:0}: Error finding container 537fd2da3c56aa22ff6a4c5fe7429a9e408971101160a6bf36d551d69c636242: Status 404 returned error can't find the container with id 537fd2da3c56aa22ff6a4c5fe7429a9e408971101160a6bf36d551d69c636242 Nov 01 06:50:09.738400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:09.738366 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" event={"ID":"85e92597-f951-4c28-b8df-0e4155eae502","Type":"ContainerStarted","Data":"c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c"} Nov 01 06:50:09.738788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:09.738430 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:09.738788 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:09.738454 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" event={"ID":"85e92597-f951-4c28-b8df-0e4155eae502","Type":"ContainerStarted","Data":"537fd2da3c56aa22ff6a4c5fe7429a9e408971101160a6bf36d551d69c636242"} Nov 01 06:50:09.854711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:09.854662 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" podStartSLOduration=1.854649805 podStartE2EDuration="1.854649805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:09.851409416 +0000 UTC m=+2425.404198609" watchObservedRunningTime="2026-11-01 06:50:09.854649805 +0000 UTC m=+2425.407438997" Nov 01 06:50:11.723805 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:11.723766 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-95e911575933: secret "agc-cred-95e911575933" not found Nov 01 06:50:11.723805 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:11.723801 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9: secret "agc-cred-95e911575933" not found Nov 01 06:50:11.724388 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:11.723854 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials podName:45a65a4a-3a31-42c3-89fe-88518f2ac9ac nodeName:}" failed. No retries permitted until 2026-11-01 06:50:12.22383977 +0000 UTC m=+2427.776628941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials") pod "payload-processing-external-model-595cb556d4-tfnm9" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac") : secret "agc-cred-95e911575933" not found Nov 01 06:50:12.229801 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:12.229772 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-95e911575933: secret "agc-cred-95e911575933" not found Nov 01 06:50:12.229801 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:12.229796 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9: secret "agc-cred-95e911575933" not found Nov 01 06:50:12.229989 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:12.229854 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials podName:45a65a4a-3a31-42c3-89fe-88518f2ac9ac nodeName:}" failed. No retries permitted until 2026-11-01 06:50:13.229839442 +0000 UTC m=+2428.782628614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials") pod "payload-processing-external-model-595cb556d4-tfnm9" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac") : secret "agc-cred-95e911575933" not found Nov 01 06:50:13.237316 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:13.237261 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-95e911575933: secret "agc-cred-95e911575933" not found Nov 01 06:50:13.237316 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:13.237309 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9: secret "agc-cred-95e911575933" not found Nov 01 06:50:13.237734 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:13.237360 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials podName:45a65a4a-3a31-42c3-89fe-88518f2ac9ac nodeName:}" failed. No retries permitted until 2026-11-01 06:50:15.237346902 +0000 UTC m=+2430.790136102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials") pod "payload-processing-external-model-595cb556d4-tfnm9" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac") : secret "agc-cred-95e911575933" not found Nov 01 06:50:13.482865 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.482833 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h"] Nov 01 06:50:13.483048 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.483024 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" podUID="b8c10ea8-b0e1-4814-84c8-dc464cf4f869" containerName="maas-api" containerID="cri-o://cb9a8ac22b70e03f0ebf095826647bad35eb1657e51c1ba4c22ce728ca3d2c90" gracePeriod=30 Nov 01 06:50:13.563168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.563105 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" containerID="cri-o://1221bb99c05a49c2a73aa3455bc13b19627899e733e4a27b6bc48cd6f54c67f7" gracePeriod=25 Nov 01 06:50:13.771024 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.770223 4914 generic.go:356] "Generic (PLEG): container finished" podID="b8c10ea8-b0e1-4814-84c8-dc464cf4f869" containerID="cb9a8ac22b70e03f0ebf095826647bad35eb1657e51c1ba4c22ce728ca3d2c90" exitCode=0 Nov 01 06:50:13.771024 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.770397 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" event={"ID":"b8c10ea8-b0e1-4814-84c8-dc464cf4f869","Type":"ContainerDied","Data":"cb9a8ac22b70e03f0ebf095826647bad35eb1657e51c1ba4c22ce728ca3d2c90"} Nov 01 06:50:13.785056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.774964 4914 generic.go:356] "Generic (PLEG): container finished" podID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerID="1221bb99c05a49c2a73aa3455bc13b19627899e733e4a27b6bc48cd6f54c67f7" exitCode=0 Nov 01 06:50:13.785056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.775071 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" event={"ID":"49e5c998-eeae-4c11-b0b8-a2ba947ee036","Type":"ContainerDied","Data":"1221bb99c05a49c2a73aa3455bc13b19627899e733e4a27b6bc48cd6f54c67f7"} Nov 01 06:50:13.931894 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.931863 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:50:13.960922 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:13.960900 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:50:14.049324 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049237 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\") pod \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " Nov 01 06:50:14.049459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049344 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume\") pod \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " Nov 01 06:50:14.049459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049373 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca\") pod \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " Nov 01 06:50:14.049459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049397 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials\") pod \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " Nov 01 06:50:14.049600 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049474 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7\") pod \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " Nov 01 06:50:14.049647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049602 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs\") pod \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " Nov 01 06:50:14.049647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049633 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume\") pod \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\" (UID: \"49e5c998-eeae-4c11-b0b8-a2ba947ee036\") " Nov 01 06:50:14.049747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049699 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff\" (UniqueName: \"kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff\") pod \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\" (UID: \"b8c10ea8-b0e1-4814-84c8-dc464cf4f869\") " Nov 01 06:50:14.049814 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.049800 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume" pod "49e5c998-eeae-4c11-b0b8-a2ba947ee036" (UID: "49e5c998-eeae-4c11-b0b8-a2ba947ee036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:14.050169 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.050142 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.050306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.050192 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume" pod "49e5c998-eeae-4c11-b0b8-a2ba947ee036" (UID: "49e5c998-eeae-4c11-b0b8-a2ba947ee036"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:14.050306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.050220 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca" pod "b8c10ea8-b0e1-4814-84c8-dc464cf4f869" (UID: "b8c10ea8-b0e1-4814-84c8-dc464cf4f869"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:14.051918 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.051891 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7" pod "49e5c998-eeae-4c11-b0b8-a2ba947ee036" (UID: "49e5c998-eeae-4c11-b0b8-a2ba947ee036"). InnerVolumeSpecName "kube-api-access-p66c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:14.052204 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.052181 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff" pod "b8c10ea8-b0e1-4814-84c8-dc464cf4f869" (UID: "b8c10ea8-b0e1-4814-84c8-dc464cf4f869"). InnerVolumeSpecName "kube-api-access-hdgff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:14.052393 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.052218 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials" pod "49e5c998-eeae-4c11-b0b8-a2ba947ee036" (UID: "49e5c998-eeae-4c11-b0b8-a2ba947ee036"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:14.052566 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.052471 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs" pod "b8c10ea8-b0e1-4814-84c8-dc464cf4f869" (UID: "b8c10ea8-b0e1-4814-84c8-dc464cf4f869"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:14.052850 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.052824 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls" pod "b8c10ea8-b0e1-4814-84c8-dc464cf4f869" (UID: "b8c10ea8-b0e1-4814-84c8-dc464cf4f869"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 06:50:14.151521 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151499 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.151521 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151520 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.151705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151529 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p66c7\" (UniqueName: \"kubernetes.io/projected/49e5c998-eeae-4c11-b0b8-a2ba947ee036-kube-api-access-p66c7\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.151705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151541 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.151705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151550 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49e5c998-eeae-4c11-b0b8-a2ba947ee036-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.151705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151558 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgff\" (UniqueName: \"kubernetes.io/projected/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-kube-api-access-hdgff\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.151705 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.151567 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c10ea8-b0e1-4814-84c8-dc464cf4f869-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:14.782372 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.782298 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" Nov 01 06:50:14.782372 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.782334 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h" event={"ID":"b8c10ea8-b0e1-4814-84c8-dc464cf4f869","Type":"ContainerDied","Data":"2ce0781f118a69da10fecc634626534df28874bbb84bcbf9b88548fa1db29150"} Nov 01 06:50:14.782869 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.782391 4914 scope.go:122] "RemoveContainer" containerID="cb9a8ac22b70e03f0ebf095826647bad35eb1657e51c1ba4c22ce728ca3d2c90" Nov 01 06:50:14.783952 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.783936 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" Nov 01 06:50:14.784056 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.783937 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj" event={"ID":"49e5c998-eeae-4c11-b0b8-a2ba947ee036","Type":"ContainerDied","Data":"ee76844ad1a41c891f8c3918f2286341d4c5f8df32f5e657372b03dcc35aa995"} Nov 01 06:50:14.793775 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.793757 4914 scope.go:122] "RemoveContainer" containerID="1221bb99c05a49c2a73aa3455bc13b19627899e733e4a27b6bc48cd6f54c67f7" Nov 01 06:50:14.945818 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.945788 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h"] Nov 01 06:50:14.994624 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:14.994600 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8a9760-65879bfc8f-8fn9h"] Nov 01 06:50:15.087582 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:15.087504 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj"] Nov 01 06:50:15.091919 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:15.091887 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-787d5c596d-gsfjj"] Nov 01 06:50:15.290230 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:15.290185 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-95e911575933: secret "agc-cred-95e911575933" not found Nov 01 06:50:15.290230 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:15.290214 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9: secret "agc-cred-95e911575933" not found Nov 01 06:50:15.290469 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:15.290318 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials podName:45a65a4a-3a31-42c3-89fe-88518f2ac9ac nodeName:}" failed. No retries permitted until 2026-11-01 06:50:19.290268415 +0000 UTC m=+2434.843057603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials") pod "payload-processing-external-model-595cb556d4-tfnm9" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac") : secret "agc-cred-95e911575933" not found Nov 01 06:50:16.619412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:16.619380 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" path="/var/lib/kubelet/pods/49e5c998-eeae-4c11-b0b8-a2ba947ee036/volumes" Nov 01 06:50:16.619822 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:16.619808 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c10ea8-b0e1-4814-84c8-dc464cf4f869" path="/var/lib/kubelet/pods/b8c10ea8-b0e1-4814-84c8-dc464cf4f869/volumes" Nov 01 06:50:19.348653 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:19.348619 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-95e911575933: secret "agc-cred-95e911575933" not found Nov 01 06:50:19.348653 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:19.348648 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9: secret "agc-cred-95e911575933" not found Nov 01 06:50:19.349098 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:19.348734 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials podName:45a65a4a-3a31-42c3-89fe-88518f2ac9ac nodeName:}" failed. No retries permitted until 2026-11-01 06:50:27.348718733 +0000 UTC m=+2442.901507908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials") pod "payload-processing-external-model-595cb556d4-tfnm9" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac") : secret "agc-cred-95e911575933" not found Nov 01 06:50:20.748219 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:20.748183 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" podUID="85e92597-f951-4c28-b8df-0e4155eae502" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:50:21.954028 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.953998 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2"] Nov 01 06:50:21.955247 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.955219 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" podUID="85e92597-f951-4c28-b8df-0e4155eae502" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:50:21.976506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.976482 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64954bd78b-zr94k"] Nov 01 06:50:21.977120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.977100 4914 cpu_manager.go:414] "containerMap: removing container" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" Nov 01 06:50:21.977120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.977118 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" Nov 01 06:50:21.977239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.977136 4914 cpu_manager.go:414] "containerMap: removing container" podUID="b8c10ea8-b0e1-4814-84c8-dc464cf4f869" containerName="maas-api" Nov 01 06:50:21.977239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.977142 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c10ea8-b0e1-4814-84c8-dc464cf4f869" containerName="maas-api" Nov 01 06:50:21.977329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.977257 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e5c998-eeae-4c11-b0b8-a2ba947ee036" containerName="payload-processing" Nov 01 06:50:21.977329 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.977266 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c10ea8-b0e1-4814-84c8-dc464cf4f869" containerName="maas-api" Nov 01 06:50:21.982088 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:21.982070 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.033811 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.033785 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bad5b4f3f7d2\"" Nov 01 06:50:22.033904 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.033785 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ba171a9860eb\"" Nov 01 06:50:22.066961 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.066890 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64954bd78b-zr94k"] Nov 01 06:50:22.078549 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.078517 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.078704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.078673 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghm5\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.078786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.078714 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.078786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.078775 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.180406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.180372 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.180559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.180513 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghm5\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.180559 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.180547 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.180637 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.180604 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.181718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.181686 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.182090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.182070 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.184411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.184383 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.237726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.237661 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghm5\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5\") pod \"payload-processing-external-model-64954bd78b-zr94k\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.296127 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.296091 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:22.668524 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.668499 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64954bd78b-zr94k"] Nov 01 06:50:22.669655 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:50:22.669623 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdaf92f3_8192_4b1a_b2a6_9e0121a9c7f8.slice/crio-9261c7f3316bde166b5d27d4e63c278d639a3f3959fef0569c452ffe714f72c4 WatchSource:0}: Error finding container 9261c7f3316bde166b5d27d4e63c278d639a3f3959fef0569c452ffe714f72c4: Status 404 returned error can't find the container with id 9261c7f3316bde166b5d27d4e63c278d639a3f3959fef0569c452ffe714f72c4 Nov 01 06:50:22.833592 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.833565 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" event={"ID":"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8","Type":"ContainerStarted","Data":"456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b"} Nov 01 06:50:22.833711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.833618 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" event={"ID":"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8","Type":"ContainerStarted","Data":"9261c7f3316bde166b5d27d4e63c278d639a3f3959fef0569c452ffe714f72c4"} Nov 01 06:50:22.833711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:22.833691 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:23.411297 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:23.411147 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" podStartSLOduration=2.41112983 podStartE2EDuration="2.41112983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:22.949316607 +0000 UTC m=+2438.502105824" watchObservedRunningTime="2026-11-01 06:50:23.41112983 +0000 UTC m=+2438.963919023" Nov 01 06:50:23.413191 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:23.413171 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx"] Nov 01 06:50:26.964231 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:26.963805 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" podUID="85e92597-f951-4c28-b8df-0e4155eae502" containerName="payload-processing" containerID="cri-o://c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c" gracePeriod=25 Nov 01 06:50:27.407821 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:27.394256 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-95e911575933: secret "agc-cred-95e911575933" not found Nov 01 06:50:27.407821 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:27.407332 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9: secret "agc-cred-95e911575933" not found Nov 01 06:50:27.407821 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:27.407422 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials podName:45a65a4a-3a31-42c3-89fe-88518f2ac9ac nodeName:}" failed. No retries permitted until 2026-11-01 06:50:43.407401088 +0000 UTC m=+2458.960190276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials") pod "payload-processing-external-model-595cb556d4-tfnm9" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac") : secret "agc-cred-95e911575933" not found Nov 01 06:50:27.534437 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.534407 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:27.613658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.612604 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume\") pod \"85e92597-f951-4c28-b8df-0e4155eae502\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " Nov 01 06:50:27.613658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.612710 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume\") pod \"85e92597-f951-4c28-b8df-0e4155eae502\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " Nov 01 06:50:27.613658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.612824 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47\") pod \"85e92597-f951-4c28-b8df-0e4155eae502\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " Nov 01 06:50:27.613658 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.612878 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials\") pod \"85e92597-f951-4c28-b8df-0e4155eae502\" (UID: \"85e92597-f951-4c28-b8df-0e4155eae502\") " Nov 01 06:50:27.614639 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.614616 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume" pod "85e92597-f951-4c28-b8df-0e4155eae502" (UID: "85e92597-f951-4c28-b8df-0e4155eae502"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:27.615152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.615129 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume" pod "85e92597-f951-4c28-b8df-0e4155eae502" (UID: "85e92597-f951-4c28-b8df-0e4155eae502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:27.616062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.616029 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials" pod "85e92597-f951-4c28-b8df-0e4155eae502" (UID: "85e92597-f951-4c28-b8df-0e4155eae502"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:27.617710 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.617666 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47" pod "85e92597-f951-4c28-b8df-0e4155eae502" (UID: "85e92597-f951-4c28-b8df-0e4155eae502"). InnerVolumeSpecName "kube-api-access-zgr47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:27.717298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.716397 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgr47\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-kube-api-access-zgr47\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:27.717298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.716446 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85e92597-f951-4c28-b8df-0e4155eae502-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:27.717298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.716462 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:27.717298 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:27.716477 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85e92597-f951-4c28-b8df-0e4155eae502-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:28.013349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.009214 4914 generic.go:356] "Generic (PLEG): container finished" podID="85e92597-f951-4c28-b8df-0e4155eae502" containerID="c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c" exitCode=0 Nov 01 06:50:28.013349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.009444 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" event={"ID":"85e92597-f951-4c28-b8df-0e4155eae502","Type":"ContainerDied","Data":"c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c"} Nov 01 06:50:28.013349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.009531 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" event={"ID":"85e92597-f951-4c28-b8df-0e4155eae502","Type":"ContainerDied","Data":"537fd2da3c56aa22ff6a4c5fe7429a9e408971101160a6bf36d551d69c636242"} Nov 01 06:50:28.013349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.009565 4914 scope.go:122] "RemoveContainer" containerID="c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c" Nov 01 06:50:28.013349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.009732 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2" Nov 01 06:50:28.039145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.039112 4914 scope.go:122] "RemoveContainer" containerID="c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c" Nov 01 06:50:28.039559 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:28.039531 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c\": container with ID starting with c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c not found: ID does not exist" containerID="c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c" Nov 01 06:50:28.039718 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.039692 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c"} err="failed to get container status \"c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c\": rpc error: code = NotFound desc = could not find container \"c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c\": container with ID starting with c0bcdd3c02e37dc4f01a4fbac960dcfe9510cc80b5921f657c4233882878c35c not found: ID does not exist" Nov 01 06:50:28.207908 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.207859 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2"] Nov 01 06:50:28.244004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.243975 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b968cbf54-hrgz2"] Nov 01 06:50:28.419401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.419353 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" podUID="1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" containerName="payload-pre-processing" containerID="cri-o://8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038" gracePeriod=25 Nov 01 06:50:28.625094 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.625054 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e92597-f951-4c28-b8df-0e4155eae502" path="/var/lib/kubelet/pods/85e92597-f951-4c28-b8df-0e4155eae502/volumes" Nov 01 06:50:28.738506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.738481 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:50:28.742444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.742422 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume\") pod \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " Nov 01 06:50:28.742577 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.742538 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7\" (UniqueName: \"kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7\") pod \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\" (UID: \"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4\") " Nov 01 06:50:28.742763 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.742741 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume" pod "1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" (UID: "1d99c13b-daec-4f3c-9a38-4e13a6c1eee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:28.743000 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.742985 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:28.744845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.744821 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7" pod "1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" (UID: "1d99c13b-daec-4f3c-9a38-4e13a6c1eee4"). InnerVolumeSpecName "kube-api-access-ns2l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:28.843780 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:28.843747 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2l7\" (UniqueName: \"kubernetes.io/projected/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4-kube-api-access-ns2l7\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:29.017857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.017781 4914 generic.go:356] "Generic (PLEG): container finished" podID="1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" containerID="8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038" exitCode=0 Nov 01 06:50:29.017857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.017817 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" event={"ID":"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4","Type":"ContainerDied","Data":"8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038"} Nov 01 06:50:29.017857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.017823 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" Nov 01 06:50:29.017857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.017857 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx" event={"ID":"1d99c13b-daec-4f3c-9a38-4e13a6c1eee4","Type":"ContainerDied","Data":"6267da35056d20984b7469e876ae4903bf3e11de61c08d647f05b848b23d1312"} Nov 01 06:50:29.018431 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.017879 4914 scope.go:122] "RemoveContainer" containerID="8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038" Nov 01 06:50:29.028888 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.028710 4914 scope.go:122] "RemoveContainer" containerID="8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038" Nov 01 06:50:29.029045 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:29.029027 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038\": container with ID starting with 8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038 not found: ID does not exist" containerID="8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038" Nov 01 06:50:29.029093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.029054 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038"} err="failed to get container status \"8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038\": rpc error: code = NotFound desc = could not find container \"8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038\": container with ID starting with 8a2fc71e675ea15aa94178aa8dbbf49ec5d593543427411040fe6ec30de22038 not found: ID does not exist" Nov 01 06:50:29.179685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.179656 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx"] Nov 01 06:50:29.186964 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:29.186938 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d855db-658cb4ffd7-rwskx"] Nov 01 06:50:30.616675 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:30.616641 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" path="/var/lib/kubelet/pods/1d99c13b-daec-4f3c-9a38-4e13a6c1eee4/volumes" Nov 01 06:50:33.412032 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.411992 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb"] Nov 01 06:50:33.412835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.412813 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" containerName="payload-pre-processing" Nov 01 06:50:33.412835 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.412835 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" containerName="payload-pre-processing" Nov 01 06:50:33.412994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.412853 4914 cpu_manager.go:414] "containerMap: removing container" podUID="85e92597-f951-4c28-b8df-0e4155eae502" containerName="payload-processing" Nov 01 06:50:33.412994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.412860 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e92597-f951-4c28-b8df-0e4155eae502" containerName="payload-processing" Nov 01 06:50:33.413093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.413045 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d99c13b-daec-4f3c-9a38-4e13a6c1eee4" containerName="payload-pre-processing" Nov 01 06:50:33.413093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.413060 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e92597-f951-4c28-b8df-0e4155eae502" containerName="payload-processing" Nov 01 06:50:33.417773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.417753 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.470167 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.470144 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-302ba0-dockercfg-4kvtl\"" Nov 01 06:50:33.507969 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.498355 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87sg\" (UniqueName: \"kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg\") pod \"payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.533308 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.529664 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.559403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.559371 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-302ba0\"" Nov 01 06:50:33.565429 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.565384 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb"] Nov 01 06:50:33.579929 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.575626 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj"] Nov 01 06:50:33.588943 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.587117 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.630985 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.630955 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.631178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.631027 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87sg\" (UniqueName: \"kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg\") pod \"payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.631713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.631686 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.634158 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.634115 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj"] Nov 01 06:50:33.687209 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.687155 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87sg\" (UniqueName: \"kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg\") pod \"payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.730987 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.730966 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:33.731829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.731802 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.731941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.731870 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsvd\" (UniqueName: \"kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd\") pod \"payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.833697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.833661 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.833857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.833715 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsvd\" (UniqueName: \"kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd\") pod \"payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.834654 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.834620 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.843622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.843601 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:50:33.889974 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.889949 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsvd\" (UniqueName: \"kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd\") pod \"payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.907615 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.907587 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:33.985786 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:33.985759 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9"] Nov 01 06:50:34.106374 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:34.106345 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb"] Nov 01 06:50:34.108334 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:50:34.108303 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68459023_3b72_4d96_840f_b7ef5ace2865.slice/crio-30e6a1d43fa48e7c76a621afc8e18d798ca6c166c3aa978092f9a942734f9028 WatchSource:0}: Error finding container 30e6a1d43fa48e7c76a621afc8e18d798ca6c166c3aa978092f9a942734f9028: Status 404 returned error can't find the container with id 30e6a1d43fa48e7c76a621afc8e18d798ca6c166c3aa978092f9a942734f9028 Nov 01 06:50:34.279239 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:34.279210 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj"] Nov 01 06:50:34.280123 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:50:34.280099 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dc3dd9_c337_49f6_82ea_3950b2c9b280.slice/crio-baac14f3dca0a2b691cdf3943dd55ceb09995c87a0a154a13f30ab7648329256 WatchSource:0}: Error finding container baac14f3dca0a2b691cdf3943dd55ceb09995c87a0a154a13f30ab7648329256: Status 404 returned error can't find the container with id baac14f3dca0a2b691cdf3943dd55ceb09995c87a0a154a13f30ab7648329256 Nov 01 06:50:35.055547 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.055509 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" event={"ID":"19dc3dd9-c337-49f6-82ea-3950b2c9b280","Type":"ContainerStarted","Data":"2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07"} Nov 01 06:50:35.055986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.055576 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:35.055986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.055596 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" event={"ID":"19dc3dd9-c337-49f6-82ea-3950b2c9b280","Type":"ContainerStarted","Data":"baac14f3dca0a2b691cdf3943dd55ceb09995c87a0a154a13f30ab7648329256"} Nov 01 06:50:35.057477 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.057447 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" event={"ID":"68459023-3b72-4d96-840f-b7ef5ace2865","Type":"ContainerStarted","Data":"8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d"} Nov 01 06:50:35.057588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.057499 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" event={"ID":"68459023-3b72-4d96-840f-b7ef5ace2865","Type":"ContainerStarted","Data":"30e6a1d43fa48e7c76a621afc8e18d798ca6c166c3aa978092f9a942734f9028"} Nov 01 06:50:35.057588 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.057532 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:50:35.169855 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.169806 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" podStartSLOduration=2.169792762 podStartE2EDuration="2.169792762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:35.167144606 +0000 UTC m=+2450.719933824" watchObservedRunningTime="2026-11-01 06:50:35.169792762 +0000 UTC m=+2450.722581954" Nov 01 06:50:35.305090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:35.303869 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" podStartSLOduration=2.303847906 podStartE2EDuration="2.303847906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:50:35.301833353 +0000 UTC m=+2450.854622546" watchObservedRunningTime="2026-11-01 06:50:35.303847906 +0000 UTC m=+2450.856637101" Nov 01 06:50:37.592074 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:37.592041 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:50:38.986594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:38.986559 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" podUID="45a65a4a-3a31-42c3-89fe-88518f2ac9ac" containerName="payload-processing" containerID="cri-o://417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c" gracePeriod=25 Nov 01 06:50:39.238949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.238891 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:50:39.389578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.389545 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume\") pod \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " Nov 01 06:50:39.389720 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.389692 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp\") pod \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " Nov 01 06:50:39.389794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.389779 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials\") pod \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " Nov 01 06:50:39.389826 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.389806 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume\") pod \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\" (UID: \"45a65a4a-3a31-42c3-89fe-88518f2ac9ac\") " Nov 01 06:50:39.390033 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.389997 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume" pod "45a65a4a-3a31-42c3-89fe-88518f2ac9ac" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:39.390294 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.390246 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume" pod "45a65a4a-3a31-42c3-89fe-88518f2ac9ac" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:50:39.390414 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.390301 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:39.391973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.391948 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp" pod "45a65a4a-3a31-42c3-89fe-88518f2ac9ac" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac"). InnerVolumeSpecName "kube-api-access-pgglp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:39.392063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.392038 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials" pod "45a65a4a-3a31-42c3-89fe-88518f2ac9ac" (UID: "45a65a4a-3a31-42c3-89fe-88518f2ac9ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:50:39.491296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.491213 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgglp\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-kube-api-access-pgglp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:39.491296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.491239 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:39.491296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:39.491249 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45a65a4a-3a31-42c3-89fe-88518f2ac9ac-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:50:40.091309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.091263 4914 generic.go:356] "Generic (PLEG): container finished" podID="45a65a4a-3a31-42c3-89fe-88518f2ac9ac" containerID="417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c" exitCode=0 Nov 01 06:50:40.091833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.091327 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" Nov 01 06:50:40.091833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.091343 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" event={"ID":"45a65a4a-3a31-42c3-89fe-88518f2ac9ac","Type":"ContainerDied","Data":"417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c"} Nov 01 06:50:40.091833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.091395 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9" event={"ID":"45a65a4a-3a31-42c3-89fe-88518f2ac9ac","Type":"ContainerDied","Data":"e323144aa1de48fb77c9edc66971a57ee2662808c2f4bb70a611b37b9e56984a"} Nov 01 06:50:40.091833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.091411 4914 scope.go:122] "RemoveContainer" containerID="417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c" Nov 01 06:50:40.101544 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.101528 4914 scope.go:122] "RemoveContainer" containerID="417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c" Nov 01 06:50:40.101791 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:50:40.101773 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c\": container with ID starting with 417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c not found: ID does not exist" containerID="417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c" Nov 01 06:50:40.101834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.101797 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c"} err="failed to get container status \"417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c\": rpc error: code = NotFound desc = could not find container \"417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c\": container with ID starting with 417f0bfce55c344e234919eeb15947a4b471aa8a6c867ce60c24cc9ca02e632c not found: ID does not exist" Nov 01 06:50:40.253682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.253646 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9"] Nov 01 06:50:40.255960 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.255940 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-595cb556d4-tfnm9"] Nov 01 06:50:40.616187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:40.616156 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a65a4a-3a31-42c3-89fe-88518f2ac9ac" path="/var/lib/kubelet/pods/45a65a4a-3a31-42c3-89fe-88518f2ac9ac/volumes" Nov 01 06:50:46.093193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:46.093156 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 06:50:46.095499 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:50:46.095418 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 06:51:06.866258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.866180 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw"] Nov 01 06:51:06.866827 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.866808 4914 cpu_manager.go:414] "containerMap: removing container" podUID="45a65a4a-3a31-42c3-89fe-88518f2ac9ac" containerName="payload-processing" Nov 01 06:51:06.866898 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.866829 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a65a4a-3a31-42c3-89fe-88518f2ac9ac" containerName="payload-processing" Nov 01 06:51:06.866970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.866961 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a65a4a-3a31-42c3-89fe-88518f2ac9ac" containerName="payload-processing" Nov 01 06:51:06.871973 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.871953 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.887112 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.887090 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxdd\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.887240 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.887129 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.887343 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.887320 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.887417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.887357 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.976511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.976487 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw"] Nov 01 06:51:06.988384 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.988358 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxdd\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.988500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.988403 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.988500 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.988489 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.988572 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.988515 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.989222 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.989167 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.989222 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.989224 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:06.991215 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:06.991193 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:07.044166 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:07.044139 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxdd\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd\") pod \"payload-processing-external-model-866d5b9c86-vpmlw\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:07.185512 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:07.185457 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:07.557700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:07.557667 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw"] Nov 01 06:51:07.558998 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:07.558966 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6fe22c_9993_4f65_8f1d_2822b0493ba5.slice/crio-7d7d8cd70ca892d96279bc1a83ca4ba3758be595dbd53e2b97300f26f8ae8a05 WatchSource:0}: Error finding container 7d7d8cd70ca892d96279bc1a83ca4ba3758be595dbd53e2b97300f26f8ae8a05: Status 404 returned error can't find the container with id 7d7d8cd70ca892d96279bc1a83ca4ba3758be595dbd53e2b97300f26f8ae8a05 Nov 01 06:51:07.592709 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:07.592680 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:51:08.283452 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:08.283410 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" event={"ID":"dc6fe22c-9993-4f65-8f1d-2822b0493ba5","Type":"ContainerStarted","Data":"3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c"} Nov 01 06:51:08.283974 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:08.283476 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" event={"ID":"dc6fe22c-9993-4f65-8f1d-2822b0493ba5","Type":"ContainerStarted","Data":"7d7d8cd70ca892d96279bc1a83ca4ba3758be595dbd53e2b97300f26f8ae8a05"} Nov 01 06:51:08.283974 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:08.283562 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:08.398187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:08.398133 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" podStartSLOduration=2.398118941 podStartE2EDuration="2.398118941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:08.39570459 +0000 UTC m=+2483.948493940" watchObservedRunningTime="2026-11-01 06:51:08.398118941 +0000 UTC m=+2483.950908133" Nov 01 06:51:11.724509 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.724472 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw"] Nov 01 06:51:11.747942 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.747909 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-ln7cv"] Nov 01 06:51:11.751935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.751917 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.803345 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.803325 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-75fbcfa535aa\"" Nov 01 06:51:11.834148 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.834089 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-ln7cv"] Nov 01 06:51:11.854205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.854174 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847cb\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.854360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.854233 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.854497 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.854473 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.854564 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.854523 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.955398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.955370 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.955526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.955410 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.955526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.955446 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847cb\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.955526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.955474 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.956077 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.956056 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.956200 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.956183 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:11.958128 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:11.958106 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:12.011846 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:12.011792 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847cb\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb\") pod \"payload-processing-external-model-985777bbf-ln7cv\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:12.065568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:12.065547 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:12.438711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:12.438687 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-ln7cv"] Nov 01 06:51:12.440065 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:12.440038 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006853e9_dd42_4827_9f29_8e308c7bc5f2.slice/crio-e7158905b0a35d828be67c24729cc7f1df01a34327e3199431ddf80aa100fae3 WatchSource:0}: Error finding container e7158905b0a35d828be67c24729cc7f1df01a34327e3199431ddf80aa100fae3: Status 404 returned error can't find the container with id e7158905b0a35d828be67c24729cc7f1df01a34327e3199431ddf80aa100fae3 Nov 01 06:51:13.316333 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:13.316292 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" event={"ID":"006853e9-dd42-4827-9f29-8e308c7bc5f2","Type":"ContainerStarted","Data":"89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab"} Nov 01 06:51:13.316725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:13.316368 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" event={"ID":"006853e9-dd42-4827-9f29-8e308c7bc5f2","Type":"ContainerStarted","Data":"e7158905b0a35d828be67c24729cc7f1df01a34327e3199431ddf80aa100fae3"} Nov 01 06:51:13.431444 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:13.431387 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" podStartSLOduration=2.431368874 podStartE2EDuration="2.431368874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:13.428824472 +0000 UTC m=+2488.981613657" watchObservedRunningTime="2026-11-01 06:51:13.431368874 +0000 UTC m=+2488.984158067" Nov 01 06:51:14.331065 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:14.331027 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:14.415591 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:14.415551 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:14.415831 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:14.415813 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-ln7cv: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:14.417317 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:14.417297 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials podName:006853e9-dd42-4827-9f29-8e308c7bc5f2 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:14.917255325 +0000 UTC m=+2490.470044508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials") pod "payload-processing-external-model-985777bbf-ln7cv" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:14.925312 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:14.925261 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:14.925551 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:14.925535 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-ln7cv: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:14.925722 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:14.925709 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials podName:006853e9-dd42-4827-9f29-8e308c7bc5f2 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:15.925689166 +0000 UTC m=+2491.478478357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials") pod "payload-processing-external-model-985777bbf-ln7cv" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:15.937153 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:15.937115 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:15.937682 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:15.937158 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-ln7cv: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:15.937682 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:15.937221 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials podName:006853e9-dd42-4827-9f29-8e308c7bc5f2 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:17.937203117 +0000 UTC m=+2493.489992306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials") pod "payload-processing-external-model-985777bbf-ln7cv" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:16.726525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.726483 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" podUID="dc6fe22c-9993-4f65-8f1d-2822b0493ba5" containerName="payload-processing" containerID="cri-o://3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c" gracePeriod=25 Nov 01 06:51:16.763728 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.763700 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-ln7cv"] Nov 01 06:51:16.805339 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.805309 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn"] Nov 01 06:51:16.809460 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.809444 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.845433 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.845407 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.845530 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.845468 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjbb\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.845580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.845530 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.845669 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.845647 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.885877 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.885845 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn"] Nov 01 06:51:16.946881 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.946830 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.947258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.947014 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.947258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.947108 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjbb\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.947258 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.947161 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.948008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.947982 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.948160 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.948118 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.950774 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.950750 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:16.992561 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:16.992512 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:17.004187 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.004167 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjbb\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb\") pod \"payload-processing-external-model-866d5b9c86-qm2hn\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:17.048195 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048173 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume\") pod \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " Nov 01 06:51:17.048310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048225 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd\") pod \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " Nov 01 06:51:17.048310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048299 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials\") pod \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " Nov 01 06:51:17.048428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048351 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume\") pod \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\" (UID: \"dc6fe22c-9993-4f65-8f1d-2822b0493ba5\") " Nov 01 06:51:17.048555 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048522 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume" pod "dc6fe22c-9993-4f65-8f1d-2822b0493ba5" (UID: "dc6fe22c-9993-4f65-8f1d-2822b0493ba5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:17.048779 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048758 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume" pod "dc6fe22c-9993-4f65-8f1d-2822b0493ba5" (UID: "dc6fe22c-9993-4f65-8f1d-2822b0493ba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:17.048873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.048779 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:17.050264 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.050243 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd" pod "dc6fe22c-9993-4f65-8f1d-2822b0493ba5" (UID: "dc6fe22c-9993-4f65-8f1d-2822b0493ba5"). InnerVolumeSpecName "kube-api-access-sdxdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:17.050406 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.050389 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials" pod "dc6fe22c-9993-4f65-8f1d-2822b0493ba5" (UID: "dc6fe22c-9993-4f65-8f1d-2822b0493ba5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:17.122809 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.122790 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:17.149873 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.149841 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxdd\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-kube-api-access-sdxdd\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:17.149993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.149880 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:17.149993 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.149897 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6fe22c-9993-4f65-8f1d-2822b0493ba5-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:17.354336 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.354299 4914 generic.go:356] "Generic (PLEG): container finished" podID="dc6fe22c-9993-4f65-8f1d-2822b0493ba5" containerID="3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c" exitCode=0 Nov 01 06:51:17.354525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.354337 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" Nov 01 06:51:17.354525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.354378 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" event={"ID":"dc6fe22c-9993-4f65-8f1d-2822b0493ba5","Type":"ContainerDied","Data":"3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c"} Nov 01 06:51:17.354525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.354439 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw" event={"ID":"dc6fe22c-9993-4f65-8f1d-2822b0493ba5","Type":"ContainerDied","Data":"7d7d8cd70ca892d96279bc1a83ca4ba3758be595dbd53e2b97300f26f8ae8a05"} Nov 01 06:51:17.354525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.354471 4914 scope.go:122] "RemoveContainer" containerID="3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c" Nov 01 06:51:17.364332 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.364316 4914 scope.go:122] "RemoveContainer" containerID="3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c" Nov 01 06:51:17.364610 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:17.364594 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c\": container with ID starting with 3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c not found: ID does not exist" containerID="3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c" Nov 01 06:51:17.364649 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.364617 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c"} err="failed to get container status \"3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c\": rpc error: code = NotFound desc = could not find container \"3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c\": container with ID starting with 3117f6eca8dcb9a37b35e8ac6442491f1134e06343994833e3c236c51d108e4c not found: ID does not exist" Nov 01 06:51:17.497731 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.497710 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn"] Nov 01 06:51:17.498662 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:17.498640 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f9b11a_9f3f_4a0e_9297_748e77124954.slice/crio-3da9a88dbe17108b484fb6bb05de5fd53f817f6c322e5cace44101fcd4e13bd2 WatchSource:0}: Error finding container 3da9a88dbe17108b484fb6bb05de5fd53f817f6c322e5cace44101fcd4e13bd2: Status 404 returned error can't find the container with id 3da9a88dbe17108b484fb6bb05de5fd53f817f6c322e5cace44101fcd4e13bd2 Nov 01 06:51:17.517545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.517524 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw"] Nov 01 06:51:17.520465 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:17.520441 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-vpmlw"] Nov 01 06:51:17.957524 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:17.957499 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-75fbcfa535aa: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:17.957524 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:17.957526 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-985777bbf-ln7cv: secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:17.957907 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:17.957578 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials podName:006853e9-dd42-4827-9f29-8e308c7bc5f2 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:21.957560191 +0000 UTC m=+2497.510349365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials") pod "payload-processing-external-model-985777bbf-ln7cv" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2") : secret "agc-cred-75fbcfa535aa" not found Nov 01 06:51:18.362879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:18.362848 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" event={"ID":"45f9b11a-9f3f-4a0e-9297-748e77124954","Type":"ContainerStarted","Data":"fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff"} Nov 01 06:51:18.363059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:18.362905 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" event={"ID":"45f9b11a-9f3f-4a0e-9297-748e77124954","Type":"ContainerStarted","Data":"3da9a88dbe17108b484fb6bb05de5fd53f817f6c322e5cace44101fcd4e13bd2"} Nov 01 06:51:18.363059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:18.363034 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:18.476098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:18.476056 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" podStartSLOduration=2.476043852 podStartE2EDuration="2.476043852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:18.475095488 +0000 UTC m=+2494.027884684" watchObservedRunningTime="2026-11-01 06:51:18.476043852 +0000 UTC m=+2494.028833044" Nov 01 06:51:18.617124 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:18.617060 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6fe22c-9993-4f65-8f1d-2822b0493ba5" path="/var/lib/kubelet/pods/dc6fe22c-9993-4f65-8f1d-2822b0493ba5/volumes" Nov 01 06:51:19.949143 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.949108 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69"] Nov 01 06:51:19.949778 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.949759 4914 cpu_manager.go:414] "containerMap: removing container" podUID="dc6fe22c-9993-4f65-8f1d-2822b0493ba5" containerName="payload-processing" Nov 01 06:51:19.949839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.949779 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6fe22c-9993-4f65-8f1d-2822b0493ba5" containerName="payload-processing" Nov 01 06:51:19.949910 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.949901 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6fe22c-9993-4f65-8f1d-2822b0493ba5" containerName="payload-processing" Nov 01 06:51:19.954520 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.954499 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:19.976686 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.976664 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:19.976809 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.976741 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:19.976889 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.976851 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:19.976966 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:19.976888 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhtp\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.010846 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.010824 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"kube-root-ca.crt\"" Nov 01 06:51:20.011177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.011143 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"payload-processing-external-model-e2e--b10087ce-dockercfg-crgnv\"" Nov 01 06:51:20.013060 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.013038 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69"] Nov 01 06:51:20.059075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.059055 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-caf74d\"" Nov 01 06:51:20.059075 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.059069 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"routing-overlay\"" Nov 01 06:51:20.059254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.059069 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"e2e-cfl-6b3232-provider-cred\"" Nov 01 06:51:20.059254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.059104 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"openshift-service-ca.crt\"" Nov 01 06:51:20.077905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.077881 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.077990 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.077908 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhtp\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.078035 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.078018 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.078111 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.078096 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.078745 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.078720 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.078801 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.078779 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.081057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.081035 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.134599 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.134576 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhtp\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.268430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.268348 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:20.646149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:20.646121 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69"] Nov 01 06:51:20.647038 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:20.647015 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3343fe53_6010_4d65_836c_94980fcfdcf4.slice/crio-ea1b07f7aac4f8e0ed4a4a8b083b1a1d65f1e26d597503c47bc2324d847bdada WatchSource:0}: Error finding container ea1b07f7aac4f8e0ed4a4a8b083b1a1d65f1e26d597503c47bc2324d847bdada: Status 404 returned error can't find the container with id ea1b07f7aac4f8e0ed4a4a8b083b1a1d65f1e26d597503c47bc2324d847bdada Nov 01 06:51:21.383475 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:21.383444 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" event={"ID":"3343fe53-6010-4d65-836c-94980fcfdcf4","Type":"ContainerStarted","Data":"326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34"} Nov 01 06:51:21.383843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:21.383502 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" event={"ID":"3343fe53-6010-4d65-836c-94980fcfdcf4","Type":"ContainerStarted","Data":"ea1b07f7aac4f8e0ed4a4a8b083b1a1d65f1e26d597503c47bc2324d847bdada"} Nov 01 06:51:21.383843 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:21.383531 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:21.497931 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:21.497886 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" podStartSLOduration=2.497872602 podStartE2EDuration="2.497872602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:21.495788019 +0000 UTC m=+2497.048577220" watchObservedRunningTime="2026-11-01 06:51:21.497872602 +0000 UTC m=+2497.050661794" Nov 01 06:51:21.764550 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:21.764468 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" podUID="006853e9-dd42-4827-9f29-8e308c7bc5f2" containerName="payload-processing" containerID="cri-o://89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab" gracePeriod=25 Nov 01 06:51:22.018605 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.018547 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:22.104114 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104092 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume\") pod \"006853e9-dd42-4827-9f29-8e308c7bc5f2\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " Nov 01 06:51:22.104226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104123 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume\") pod \"006853e9-dd42-4827-9f29-8e308c7bc5f2\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " Nov 01 06:51:22.104226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104176 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb\") pod \"006853e9-dd42-4827-9f29-8e308c7bc5f2\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " Nov 01 06:51:22.104226 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104208 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials\") pod \"006853e9-dd42-4827-9f29-8e308c7bc5f2\" (UID: \"006853e9-dd42-4827-9f29-8e308c7bc5f2\") " Nov 01 06:51:22.104491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104465 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume" pod "006853e9-dd42-4827-9f29-8e308c7bc5f2" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:22.104622 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104600 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume" pod "006853e9-dd42-4827-9f29-8e308c7bc5f2" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:22.104706 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.104687 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:22.106473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.106450 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials" pod "006853e9-dd42-4827-9f29-8e308c7bc5f2" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:22.106473 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.106461 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb" pod "006853e9-dd42-4827-9f29-8e308c7bc5f2" (UID: "006853e9-dd42-4827-9f29-8e308c7bc5f2"). InnerVolumeSpecName "kube-api-access-847cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:22.206230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.206200 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006853e9-dd42-4827-9f29-8e308c7bc5f2-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:22.206230 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.206225 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847cb\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-kube-api-access-847cb\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:22.206426 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.206237 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/006853e9-dd42-4827-9f29-8e308c7bc5f2-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:22.390804 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.390773 4914 generic.go:356] "Generic (PLEG): container finished" podID="006853e9-dd42-4827-9f29-8e308c7bc5f2" containerID="89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab" exitCode=0 Nov 01 06:51:22.391208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.390806 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" event={"ID":"006853e9-dd42-4827-9f29-8e308c7bc5f2","Type":"ContainerDied","Data":"89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab"} Nov 01 06:51:22.391208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.390831 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" Nov 01 06:51:22.391208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.390867 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-985777bbf-ln7cv" event={"ID":"006853e9-dd42-4827-9f29-8e308c7bc5f2","Type":"ContainerDied","Data":"e7158905b0a35d828be67c24729cc7f1df01a34327e3199431ddf80aa100fae3"} Nov 01 06:51:22.391208 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.390897 4914 scope.go:122] "RemoveContainer" containerID="89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab" Nov 01 06:51:22.408623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.408598 4914 scope.go:122] "RemoveContainer" containerID="89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab" Nov 01 06:51:22.408949 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:22.408926 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab\": container with ID starting with 89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab not found: ID does not exist" containerID="89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab" Nov 01 06:51:22.408996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.408958 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab"} err="failed to get container status \"89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab\": rpc error: code = NotFound desc = could not find container \"89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab\": container with ID starting with 89c6b41f53478a5d197d06347000dd98a2a4d8b0f076bcf58a0468447f6a7eab not found: ID does not exist" Nov 01 06:51:22.553145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.553118 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-ln7cv"] Nov 01 06:51:22.602673 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.602647 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-985777bbf-ln7cv"] Nov 01 06:51:22.617557 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:22.617533 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006853e9-dd42-4827-9f29-8e308c7bc5f2" path="/var/lib/kubelet/pods/006853e9-dd42-4827-9f29-8e308c7bc5f2/volumes" Nov 01 06:51:26.906853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:26.906810 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn"] Nov 01 06:51:26.909425 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:26.909396 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" podUID="45f9b11a-9f3f-4a0e-9297-748e77124954" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:51:30.291251 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.291208 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c"] Nov 01 06:51:30.292050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.292027 4914 cpu_manager.go:414] "containerMap: removing container" podUID="006853e9-dd42-4827-9f29-8e308c7bc5f2" containerName="payload-processing" Nov 01 06:51:30.292163 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.292052 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006853e9-dd42-4827-9f29-8e308c7bc5f2" containerName="payload-processing" Nov 01 06:51:30.292223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.292198 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="006853e9-dd42-4827-9f29-8e308c7bc5f2" containerName="payload-processing" Nov 01 06:51:30.295808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.295787 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.406008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.405975 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-302ba0-serving-cert\"" Nov 01 06:51:30.409583 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.409557 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c"] Nov 01 06:51:30.452256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.452224 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.452441 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.452284 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.452441 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.452352 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46xs\" (UniqueName: \"kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.452525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.452452 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.556105 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.556006 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.556412 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.556385 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.556593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.556576 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46xs\" (UniqueName: \"kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.556849 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.556819 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.557225 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:30.557206 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-serving-cert: secret "maas-api-e2e-shared-a-w0-302ba0-serving-cert" not found Nov 01 06:51:30.566977 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:30.566942 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls podName:e5f792e6-c90a-43b4-b9a0-a5e01439eb7b nodeName:}" failed. No retries permitted until 2026-11-01 06:51:31.066913082 +0000 UTC m=+2506.619702258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls") pod "maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" (UID: "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b") : secret "maas-api-e2e-shared-a-w0-302ba0-serving-cert" not found Nov 01 06:51:30.567236 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.564819 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.568323 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.568261 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:30.621771 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:30.621740 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46xs\" (UniqueName: \"kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:31.167632 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:31.167594 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:31.170193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:31.170168 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:31.209134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:31.209106 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:31.582004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:31.581976 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c"] Nov 01 06:51:31.583455 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:31.583425 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f792e6_c90a_43b4_b9a0_a5e01439eb7b.slice/crio-b8fd092a1f533baf6c87209c9c46e74491578ff48b0a8d7800b6c35961c90696 WatchSource:0}: Error finding container b8fd092a1f533baf6c87209c9c46e74491578ff48b0a8d7800b6c35961c90696: Status 404 returned error can't find the container with id b8fd092a1f533baf6c87209c9c46e74491578ff48b0a8d7800b6c35961c90696 Nov 01 06:51:31.584682 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:31.584666 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:51:31.908309 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:31.907605 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" podUID="45f9b11a-9f3f-4a0e-9297-748e77124954" containerName="payload-processing" containerID="cri-o://fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff" gracePeriod=25 Nov 01 06:51:32.220342 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.220312 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:32.383449 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.383426 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume\") pod \"45f9b11a-9f3f-4a0e-9297-748e77124954\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " Nov 01 06:51:32.383578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.383556 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb\") pod \"45f9b11a-9f3f-4a0e-9297-748e77124954\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " Nov 01 06:51:32.383619 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.383593 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume\") pod \"45f9b11a-9f3f-4a0e-9297-748e77124954\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " Nov 01 06:51:32.383619 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.383612 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials\") pod \"45f9b11a-9f3f-4a0e-9297-748e77124954\" (UID: \"45f9b11a-9f3f-4a0e-9297-748e77124954\") " Nov 01 06:51:32.383880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.383856 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume" pod "45f9b11a-9f3f-4a0e-9297-748e77124954" (UID: "45f9b11a-9f3f-4a0e-9297-748e77124954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:32.383979 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.383943 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume" pod "45f9b11a-9f3f-4a0e-9297-748e77124954" (UID: "45f9b11a-9f3f-4a0e-9297-748e77124954"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:51:32.385860 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.385839 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials" pod "45f9b11a-9f3f-4a0e-9297-748e77124954" (UID: "45f9b11a-9f3f-4a0e-9297-748e77124954"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:32.385860 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.385849 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb" pod "45f9b11a-9f3f-4a0e-9297-748e77124954" (UID: "45f9b11a-9f3f-4a0e-9297-748e77124954"). InnerVolumeSpecName "kube-api-access-9mjbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:51:32.394949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.394932 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:51:32.457996 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.457919 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" event={"ID":"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b","Type":"ContainerStarted","Data":"b8fd092a1f533baf6c87209c9c46e74491578ff48b0a8d7800b6c35961c90696"} Nov 01 06:51:32.459429 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.459405 4914 generic.go:356] "Generic (PLEG): container finished" podID="45f9b11a-9f3f-4a0e-9297-748e77124954" containerID="fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff" exitCode=0 Nov 01 06:51:32.459529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.459441 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" event={"ID":"45f9b11a-9f3f-4a0e-9297-748e77124954","Type":"ContainerDied","Data":"fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff"} Nov 01 06:51:32.459529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.459453 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" Nov 01 06:51:32.459529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.459472 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn" event={"ID":"45f9b11a-9f3f-4a0e-9297-748e77124954","Type":"ContainerDied","Data":"3da9a88dbe17108b484fb6bb05de5fd53f817f6c322e5cace44101fcd4e13bd2"} Nov 01 06:51:32.459529 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.459493 4914 scope.go:122] "RemoveContainer" containerID="fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff" Nov 01 06:51:32.468955 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.468934 4914 scope.go:122] "RemoveContainer" containerID="fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff" Nov 01 06:51:32.469222 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:32.469201 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff\": container with ID starting with fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff not found: ID does not exist" containerID="fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff" Nov 01 06:51:32.469266 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.469229 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff"} err="failed to get container status \"fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff\": rpc error: code = NotFound desc = could not find container \"fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff\": container with ID starting with fbaa4802568d8797cf95109ea1941d7486bc8bf23d5b6ad9e5dfaef6de670cff not found: ID does not exist" Nov 01 06:51:32.485100 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.485072 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjbb\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-kube-api-access-9mjbb\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:32.485100 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.485098 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:32.485235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.485114 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45f9b11a-9f3f-4a0e-9297-748e77124954-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:32.485235 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.485155 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f9b11a-9f3f-4a0e-9297-748e77124954-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:51:32.630442 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.630414 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn"] Nov 01 06:51:32.634854 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:32.634826 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d5b9c86-qm2hn"] Nov 01 06:51:33.467096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:33.467054 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" event={"ID":"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b","Type":"ContainerStarted","Data":"38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57"} Nov 01 06:51:33.467307 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:33.467265 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:33.581593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:33.581540 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" podStartSLOduration=2.677777652 podStartE2EDuration="3.581527274s" totalImagesPullingTime="903.749622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:33.580637053 +0000 UTC m=+2509.133426245" watchObservedRunningTime="2026-11-01 06:51:33.581527274 +0000 UTC m=+2509.134316466" Nov 01 06:51:34.616879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:34.616850 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f9b11a-9f3f-4a0e-9297-748e77124954" path="/var/lib/kubelet/pods/45f9b11a-9f3f-4a0e-9297-748e77124954/volumes" Nov 01 06:51:37.591660 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:37.591622 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:51:39.479683 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:39.479649 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 06:51:40.708769 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:40.708740 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:51:40.708769 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:40.708770 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:51:40.709309 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:40.708851 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:41.208825451 +0000 UTC m=+2516.761614640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:51:41.231298 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:41.225439 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:51:41.231298 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:41.225478 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:51:41.231298 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:41.225547 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:42.225528724 +0000 UTC m=+2517.778317912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:51:42.238551 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:42.238522 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:51:42.238551 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:42.238553 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:51:42.238961 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:42.238619 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:44.238602728 +0000 UTC m=+2519.791391914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:51:44.279069 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:44.279036 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:51:44.279069 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:44.279067 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:51:44.279563 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:44.279111 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:48.279099853 +0000 UTC m=+2523.831889025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:51:48.330294 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:48.330234 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:51:48.330294 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:48.330294 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:51:48.330729 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:48.330354 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:51:56.330339468 +0000 UTC m=+2531.883128639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:51:51.906921 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:51.906887 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt"] Nov 01 06:51:51.910986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:51.910940 4914 cpu_manager.go:414] "containerMap: removing container" podUID="45f9b11a-9f3f-4a0e-9297-748e77124954" containerName="payload-processing" Nov 01 06:51:51.910986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:51.910971 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f9b11a-9f3f-4a0e-9297-748e77124954" containerName="payload-processing" Nov 01 06:51:51.911205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:51.911118 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f9b11a-9f3f-4a0e-9297-748e77124954" containerName="payload-processing" Nov 01 06:51:51.922929 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:51.922906 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:51.970716 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:51.970676 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt"] Nov 01 06:51:52.017103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.017068 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a3e841\"" Nov 01 06:51:52.017103 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.017073 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a3e841-dockercfg-nfgxw\"" Nov 01 06:51:52.084332 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.084305 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume\") pod \"payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.084466 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.084337 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdrr\" (UniqueName: \"kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr\") pod \"payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.185363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.185296 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume\") pod \"payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.185363 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.185334 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdrr\" (UniqueName: \"kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr\") pod \"payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.185982 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.185959 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume\") pod \"payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.244172 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.244142 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdrr\" (UniqueName: \"kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr\") pod \"payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.542123 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.542048 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:52.915697 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:52.915672 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt"] Nov 01 06:51:52.916896 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:52.916870 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d5d3cc_966f_4d8a_8409_ef15d0697778.slice/crio-a15d3bbf2b9134f9a4f1aaefc748b30a0eabd03c4144353da4bab5e76a41e9a2 WatchSource:0}: Error finding container a15d3bbf2b9134f9a4f1aaefc748b30a0eabd03c4144353da4bab5e76a41e9a2: Status 404 returned error can't find the container with id a15d3bbf2b9134f9a4f1aaefc748b30a0eabd03c4144353da4bab5e76a41e9a2 Nov 01 06:51:53.640542 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:53.640506 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" event={"ID":"52d5d3cc-966f-4d8a-8409-ef15d0697778","Type":"ContainerStarted","Data":"98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918"} Nov 01 06:51:53.640722 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:53.640555 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" event={"ID":"52d5d3cc-966f-4d8a-8409-ef15d0697778","Type":"ContainerStarted","Data":"a15d3bbf2b9134f9a4f1aaefc748b30a0eabd03c4144353da4bab5e76a41e9a2"} Nov 01 06:51:53.640722 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:53.640626 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:51:53.754111 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:53.754060 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" podStartSLOduration=2.75404766 podStartE2EDuration="2.75404766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:53.751741445 +0000 UTC m=+2529.304530639" watchObservedRunningTime="2026-11-01 06:51:53.75404766 +0000 UTC m=+2529.306836852" Nov 01 06:51:54.508415 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.508378 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96"] Nov 01 06:51:54.512727 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.512709 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536058 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536034 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536151 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85b9c\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536208 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536358 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536339 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536371 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536514 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536489 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536715 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536694 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536851 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536826 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.536974 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.536958 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.564753 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.564725 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-302ba0-openshift-default-dockercfg-gh4kk\"" Nov 01 06:51:54.617645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.617617 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96"] Nov 01 06:51:54.638035 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638009 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85b9c\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638072 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638121 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638168 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638143 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638360 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638314 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638356 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638386 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638427 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638413 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638504 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638645 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638619 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638747 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638722 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.638962 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.638938 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.639086 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.639069 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.639365 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.639346 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.640866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.640846 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.641149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.641132 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.693825 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.693801 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.693914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.693808 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85b9c\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c\") pod \"e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:54.825430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:54.825375 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:55.248638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.248612 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96"] Nov 01 06:51:55.249896 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:51:55.249871 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5849375a_a8cc_4408_a9ec_962db5ebe542.slice/crio-c32f91b9d16093c6ca1e9cf171a8eed9bb43e8918b545d8c721fe62814bb80a0 WatchSource:0}: Error finding container c32f91b9d16093c6ca1e9cf171a8eed9bb43e8918b545d8c721fe62814bb80a0: Status 404 returned error can't find the container with id c32f91b9d16093c6ca1e9cf171a8eed9bb43e8918b545d8c721fe62814bb80a0 Nov 01 06:51:55.252390 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.252359 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:51:55.252451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.252424 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:51:55.252491 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.252451 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:51:55.655145 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.655107 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" event={"ID":"5849375a-a8cc-4408-a9ec-962db5ebe542","Type":"ContainerStarted","Data":"f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732"} Nov 01 06:51:55.655638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.655173 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" event={"ID":"5849375a-a8cc-4408-a9ec-962db5ebe542","Type":"ContainerStarted","Data":"c32f91b9d16093c6ca1e9cf171a8eed9bb43e8918b545d8c721fe62814bb80a0"} Nov 01 06:51:55.826378 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.826347 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:51:55.873678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:55.873616 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podStartSLOduration=1.87360163 podStartE2EDuration="1.87360163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:51:55.82145238 +0000 UTC m=+2531.374241572" watchObservedRunningTime="2026-11-01 06:51:55.87360163 +0000 UTC m=+2531.426390855" Nov 01 06:51:56.375148 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:56.375108 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:51:56.375148 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:56.375142 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:51:56.375416 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:51:56.375205 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:52:12.37518993 +0000 UTC m=+2547.927979115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:51:56.828027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:56.827948 4914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:51:56.828027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:56.828001 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:51:57.826608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:57.826577 4914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:51:57.826758 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:57.826627 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:51:58.826362 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:58.826328 4914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:51:58.826740 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:58.826389 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:51:59.826564 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:59.826532 4914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:51:59.826959 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:51:59.826598 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:52:00.826425 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:00.826389 4914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 01 06:52:00.826608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:00.826457 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 01 06:52:01.830616 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:01.830585 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:52:01.831041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:01.831023 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:52:01.831586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:01.831563 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 06:52:04.649431 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:04.649402 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:52:07.592369 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:07.592332 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:52:12.445718 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:12.445690 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:52:12.445718 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:12.445717 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:52:12.446121 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:12.445795 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:52:44.445781175 +0000 UTC m=+2579.998570346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:52:37.593015 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:37.592981 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:52:44.510201 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:44.510157 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:52:44.510201 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:44.510206 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:52:44.510780 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:44.510289 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:53:48.510250253 +0000 UTC m=+2644.063039443 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:52:53.608260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.608218 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69"] Nov 01 06:52:53.668970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.668932 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw"] Nov 01 06:52:53.673581 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.673561 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.726159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.726136 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6ba6e5\"" Nov 01 06:52:53.726319 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.726298 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6ba6e5-dockercfg-ktf5p\"" Nov 01 06:52:53.730220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.730194 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw"] Nov 01 06:52:53.810878 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.810843 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.811185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.811159 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjjs\" (UniqueName: \"kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs\") pod \"payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.912893 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.912803 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.912893 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.912872 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjjs\" (UniqueName: \"kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs\") pod \"payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.913771 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.913747 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.971866 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.971836 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjjs\" (UniqueName: \"kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs\") pod \"payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:53.987515 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:53.987493 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:54.368213 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:54.368131 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw"] Nov 01 06:52:55.039856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:55.039815 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" event={"ID":"50f1aff2-ac8c-4d8e-a230-3ca057da816e","Type":"ContainerStarted","Data":"8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e"} Nov 01 06:52:55.040411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:55.039880 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" event={"ID":"50f1aff2-ac8c-4d8e-a230-3ca057da816e","Type":"ContainerStarted","Data":"c96ff1abbe40cd297964e79489456b01b702b41c4473b27443135e71beb575e4"} Nov 01 06:52:55.040800 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:55.040773 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:52:55.157188 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:55.157126 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" podStartSLOduration=2.157103765 podStartE2EDuration="2.157103765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:52:55.153502106 +0000 UTC m=+2590.706291299" watchObservedRunningTime="2026-11-01 06:52:55.157103765 +0000 UTC m=+2590.709892958" Nov 01 06:52:58.612847 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:58.612795 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" podUID="3343fe53-6010-4d65-836c-94980fcfdcf4" containerName="payload-processing" containerID="cri-o://326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34" gracePeriod=25 Nov 01 06:52:58.867617 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:58.867566 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:52:59.017845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.010464 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials\") pod \"3343fe53-6010-4d65-836c-94980fcfdcf4\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " Nov 01 06:52:59.017845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.010532 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume\") pod \"3343fe53-6010-4d65-836c-94980fcfdcf4\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " Nov 01 06:52:59.017845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.010618 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp\") pod \"3343fe53-6010-4d65-836c-94980fcfdcf4\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " Nov 01 06:52:59.017845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.010702 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume\") pod \"3343fe53-6010-4d65-836c-94980fcfdcf4\" (UID: \"3343fe53-6010-4d65-836c-94980fcfdcf4\") " Nov 01 06:52:59.017845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.011497 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume" pod "3343fe53-6010-4d65-836c-94980fcfdcf4" (UID: "3343fe53-6010-4d65-836c-94980fcfdcf4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:52:59.017845 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.012184 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume" pod "3343fe53-6010-4d65-836c-94980fcfdcf4" (UID: "3343fe53-6010-4d65-836c-94980fcfdcf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:52:59.018574 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.018532 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp" pod "3343fe53-6010-4d65-836c-94980fcfdcf4" (UID: "3343fe53-6010-4d65-836c-94980fcfdcf4"). InnerVolumeSpecName "kube-api-access-5zhtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:52:59.019382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.019351 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials" pod "3343fe53-6010-4d65-836c-94980fcfdcf4" (UID: "3343fe53-6010-4d65-836c-94980fcfdcf4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:52:59.072064 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.072034 4914 generic.go:356] "Generic (PLEG): container finished" podID="3343fe53-6010-4d65-836c-94980fcfdcf4" containerID="326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34" exitCode=0 Nov 01 06:52:59.072177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.072078 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" Nov 01 06:52:59.072177 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.072082 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" event={"ID":"3343fe53-6010-4d65-836c-94980fcfdcf4","Type":"ContainerDied","Data":"326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34"} Nov 01 06:52:59.072254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.072199 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69" event={"ID":"3343fe53-6010-4d65-836c-94980fcfdcf4","Type":"ContainerDied","Data":"ea1b07f7aac4f8e0ed4a4a8b083b1a1d65f1e26d597503c47bc2324d847bdada"} Nov 01 06:52:59.072254 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.072226 4914 scope.go:122] "RemoveContainer" containerID="326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34" Nov 01 06:52:59.081925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.081910 4914 scope.go:122] "RemoveContainer" containerID="326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34" Nov 01 06:52:59.082178 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:52:59.082162 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34\": container with ID starting with 326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34 not found: ID does not exist" containerID="326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34" Nov 01 06:52:59.082225 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.082188 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34"} err="failed to get container status \"326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34\": rpc error: code = NotFound desc = could not find container \"326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34\": container with ID starting with 326648bf2aae8be6d56be146f0f8f9f2aa1bae2846e2b0abddf5d3977bb34e34 not found: ID does not exist" Nov 01 06:52:59.111976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.111946 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:59.111976 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.111975 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:59.112149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.111992 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3343fe53-6010-4d65-836c-94980fcfdcf4-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:59.112149 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.112005 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhtp\" (UniqueName: \"kubernetes.io/projected/3343fe53-6010-4d65-836c-94980fcfdcf4-kube-api-access-5zhtp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:52:59.234665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.234565 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69"] Nov 01 06:52:59.237925 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:52:59.237903 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9dhk69"] Nov 01 06:53:00.616260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:53:00.616223 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3343fe53-6010-4d65-836c-94980fcfdcf4" path="/var/lib/kubelet/pods/3343fe53-6010-4d65-836c-94980fcfdcf4/volumes" Nov 01 06:53:07.062729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:53:07.062698 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 06:53:07.631464 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:53:07.631334 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:53:37.592531 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:53:37.592490 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:53:48.525395 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:53:48.525364 4914 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6883279a9766: secret "agc-cred-6883279a9766" not found Nov 01 06:53:48.525395 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:53:48.525397 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64954bd78b-zr94k: secret "agc-cred-6883279a9766" not found Nov 01 06:53:48.525796 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:53:48.525457 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials podName:bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8 nodeName:}" failed. No retries permitted until 2026-11-01 06:55:50.525442725 +0000 UTC m=+2766.078231929 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials") pod "payload-processing-external-model-64954bd78b-zr94k" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8") : secret "agc-cred-6883279a9766" not found Nov 01 06:54:00.568370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.568291 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4"] Nov 01 06:54:00.569010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.568990 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3343fe53-6010-4d65-836c-94980fcfdcf4" containerName="payload-processing" Nov 01 06:54:00.569057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.569012 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3343fe53-6010-4d65-836c-94980fcfdcf4" containerName="payload-processing" Nov 01 06:54:00.569128 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.569119 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3343fe53-6010-4d65-836c-94980fcfdcf4" containerName="payload-processing" Nov 01 06:54:00.573725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.573709 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.626241 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.626212 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6ba6e5-serving-cert\"" Nov 01 06:54:00.645879 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.645851 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.646025 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.645911 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntz8z\" (UniqueName: \"kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.646025 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.645943 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.646117 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.646094 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.678719 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.678693 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4"] Nov 01 06:54:00.747797 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.747761 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.747950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.747901 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.748013 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.747965 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntz8z\" (UniqueName: \"kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.748069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.748030 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.748126 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:54:00.748087 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-serving-cert: secret "maas-api-e2e-shared-b-w0-6ba6e5-serving-cert" not found Nov 01 06:54:00.748181 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:54:00.748162 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls podName:c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c nodeName:}" failed. No retries permitted until 2026-11-01 06:54:01.248139822 +0000 UTC m=+2656.800928994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls") pod "maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" (UID: "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c") : secret "maas-api-e2e-shared-b-w0-6ba6e5-serving-cert" not found Nov 01 06:54:00.748513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.748494 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.750668 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.750645 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:00.805690 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:00.805667 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntz8z\" (UniqueName: \"kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:01.256796 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:01.256763 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:01.259550 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:01.259520 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:01.487001 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:01.486971 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:01.861649 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:01.861623 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4"] Nov 01 06:54:01.862258 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:54:01.862236 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a94aea_9db5_47b5_9ee3_d9ec801b1a7c.slice/crio-bf7afd502bd850e09031db68b7f27367ee132a3b6b2c94de639b4a6c7e3705b7 WatchSource:0}: Error finding container bf7afd502bd850e09031db68b7f27367ee132a3b6b2c94de639b4a6c7e3705b7: Status 404 returned error can't find the container with id bf7afd502bd850e09031db68b7f27367ee132a3b6b2c94de639b4a6c7e3705b7 Nov 01 06:54:02.462111 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:02.462077 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" event={"ID":"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c","Type":"ContainerStarted","Data":"bf7afd502bd850e09031db68b7f27367ee132a3b6b2c94de639b4a6c7e3705b7"} Nov 01 06:54:03.470631 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:03.470594 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" event={"ID":"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c","Type":"ContainerStarted","Data":"5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5"} Nov 01 06:54:03.471072 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:03.470720 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:03.587711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:03.587663 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" podStartSLOduration=2.636987594 podStartE2EDuration="3.587649004s" totalImagesPullingTime="950.66141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:03.586171602 +0000 UTC m=+2659.138960806" watchObservedRunningTime="2026-11-01 06:54:03.587649004 +0000 UTC m=+2659.140438226" Nov 01 06:54:07.592197 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:54:07.592149 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:54:09.482449 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:09.482419 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 06:54:37.592411 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:54:37.592377 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:54:37.741104 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.741074 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf"] Nov 01 06:54:37.745656 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.745602 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.796760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.796733 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.796885 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.796781 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.796949 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.796926 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ltg\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.796992 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.796980 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.850134 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.850100 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf"] Nov 01 06:54:37.898189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.898157 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ltg\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.898359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.898216 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.898432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.898360 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.898432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.898418 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.899094 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.899074 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.899165 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.899115 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.900785 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.900757 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:37.954249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:37.954230 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ltg\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg\") pod \"payload-processing-external-model-66c9f9b6bc-q27lf\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:38.059397 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:38.059344 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:38.435731 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:38.435703 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf"] Nov 01 06:54:38.436555 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:54:38.436529 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbb8af9_852b_4bc7_a67d_05c3fab75b7b.slice/crio-b7ad19f359cbb2f2dd44c0e6642b6ee7293076e49f82f644326923d544d48444 WatchSource:0}: Error finding container b7ad19f359cbb2f2dd44c0e6642b6ee7293076e49f82f644326923d544d48444: Status 404 returned error can't find the container with id b7ad19f359cbb2f2dd44c0e6642b6ee7293076e49f82f644326923d544d48444 Nov 01 06:54:38.677798 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:38.677770 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" event={"ID":"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b","Type":"ContainerStarted","Data":"e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211"} Nov 01 06:54:38.678141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:38.677809 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" event={"ID":"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b","Type":"ContainerStarted","Data":"b7ad19f359cbb2f2dd44c0e6642b6ee7293076e49f82f644326923d544d48444"} Nov 01 06:54:38.678141 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:38.677898 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:38.793079 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:38.792995 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" podStartSLOduration=1.792981885 podStartE2EDuration="1.792981885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:54:38.790761998 +0000 UTC m=+2694.343551192" watchObservedRunningTime="2026-11-01 06:54:38.792981885 +0000 UTC m=+2694.345771116" Nov 01 06:54:44.932954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:44.932916 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:54:44.940536 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:44.940509 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:54:49.686896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:49.686862 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:51.165373 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:51.165342 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf"] Nov 01 06:54:51.166700 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:51.166621 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 06:54:51.222026 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:51.221993 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64954bd78b-zr94k"] Nov 01 06:54:56.165623 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.165579 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerName="payload-processing" containerID="cri-o://e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211" gracePeriod=25 Nov 01 06:54:56.223074 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.223032 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" podUID="bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" containerName="payload-processing" containerID="cri-o://456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b" gracePeriod=25 Nov 01 06:54:56.514769 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.514744 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:56.518340 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.518317 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:54:56.588853 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.588830 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume\") pod \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " Nov 01 06:54:56.588954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.588890 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume\") pod \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " Nov 01 06:54:56.588954 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.588946 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume\") pod \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " Nov 01 06:54:56.589063 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.588975 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg\") pod \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " Nov 01 06:54:56.589120 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589092 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials\") pod \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " Nov 01 06:54:56.589182 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589144 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials\") pod \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " Nov 01 06:54:56.589237 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589187 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5\") pod \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\" (UID: \"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8\") " Nov 01 06:54:56.589371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589347 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume" pod "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:54:56.589371 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589353 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume" pod "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" (UID: "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:54:56.589511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589381 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume" pod "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:54:56.589511 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589406 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume\") pod \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\" (UID: \"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b\") " Nov 01 06:54:56.589875 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589851 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.589980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589877 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.589980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589882 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume" pod "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" (UID: "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:54:56.589980 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.589892 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.591312 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.591288 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg" pod "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" (UID: "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b"). InnerVolumeSpecName "kube-api-access-v2ltg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:54:56.591457 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.591436 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials" pod "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" (UID: "2bbb8af9-852b-4bc7-a67d-05c3fab75b7b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:54:56.591863 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.591840 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials" pod "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:54:56.591863 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.591856 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5" pod "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" (UID: "bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8"). InnerVolumeSpecName "kube-api-access-5ghm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:54:56.690926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.690862 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ltg\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-kube-api-access-v2ltg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.690926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.690893 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.690926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.690906 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.690926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.690915 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ghm5\" (UniqueName: \"kubernetes.io/projected/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8-kube-api-access-5ghm5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.690926 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.690924 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:54:56.862421 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.862385 4914 generic.go:356] "Generic (PLEG): container finished" podID="bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" containerID="456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b" exitCode=0 Nov 01 06:54:56.862611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.862435 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" Nov 01 06:54:56.862611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.862467 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" event={"ID":"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8","Type":"ContainerDied","Data":"456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b"} Nov 01 06:54:56.862611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.862553 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64954bd78b-zr94k" event={"ID":"bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8","Type":"ContainerDied","Data":"9261c7f3316bde166b5d27d4e63c278d639a3f3959fef0569c452ffe714f72c4"} Nov 01 06:54:56.862611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.862581 4914 scope.go:122] "RemoveContainer" containerID="456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b" Nov 01 06:54:56.865069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.865011 4914 generic.go:356] "Generic (PLEG): container finished" podID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerID="e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211" exitCode=0 Nov 01 06:54:56.865069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.865064 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" event={"ID":"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b","Type":"ContainerDied","Data":"e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211"} Nov 01 06:54:56.865242 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.865090 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" Nov 01 06:54:56.865352 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.865186 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf" event={"ID":"2bbb8af9-852b-4bc7-a67d-05c3fab75b7b","Type":"ContainerDied","Data":"b7ad19f359cbb2f2dd44c0e6642b6ee7293076e49f82f644326923d544d48444"} Nov 01 06:54:56.874016 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.873997 4914 scope.go:122] "RemoveContainer" containerID="456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b" Nov 01 06:54:56.874313 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:54:56.874267 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b\": container with ID starting with 456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b not found: ID does not exist" containerID="456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b" Nov 01 06:54:56.874395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.874321 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b"} err="failed to get container status \"456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b\": rpc error: code = NotFound desc = could not find container \"456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b\": container with ID starting with 456cc3afeb39389a61f2accff57154ac83edd1790fe11bf0068d7f677ae8433b not found: ID does not exist" Nov 01 06:54:56.874395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.874338 4914 scope.go:122] "RemoveContainer" containerID="e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211" Nov 01 06:54:56.882893 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.882871 4914 scope.go:122] "RemoveContainer" containerID="e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211" Nov 01 06:54:56.883135 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:54:56.883116 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211\": container with ID starting with e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211 not found: ID does not exist" containerID="e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211" Nov 01 06:54:56.883185 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:56.883141 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211"} err="failed to get container status \"e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211\": rpc error: code = NotFound desc = could not find container \"e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211\": container with ID starting with e3a66a37a25da8a52ad8c74b50cf926f81d725ea9a52515bb4543c381c505211 not found: ID does not exist" Nov 01 06:54:57.034062 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:57.033997 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf"] Nov 01 06:54:57.038129 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:57.038101 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66c9f9b6bc-q27lf"] Nov 01 06:54:57.146247 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:57.146213 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64954bd78b-zr94k"] Nov 01 06:54:57.149842 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:57.149818 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64954bd78b-zr94k"] Nov 01 06:54:58.617687 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:58.617654 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" path="/var/lib/kubelet/pods/2bbb8af9-852b-4bc7-a67d-05c3fab75b7b/volumes" Nov 01 06:54:58.618061 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:54:58.618050 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" path="/var/lib/kubelet/pods/bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8/volumes" Nov 01 06:55:07.592567 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:07.592532 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:55:18.408223 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.408176 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx"] Nov 01 06:55:18.409178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.409154 4914 cpu_manager.go:414] "containerMap: removing container" podUID="bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" containerName="payload-processing" Nov 01 06:55:18.409178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.409178 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" containerName="payload-processing" Nov 01 06:55:18.409400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.409205 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerName="payload-processing" Nov 01 06:55:18.409400 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.409212 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerName="payload-processing" Nov 01 06:55:18.409506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.409416 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdaf92f3-8192-4b1a-b2a6-9e0121a9c7f8" containerName="payload-processing" Nov 01 06:55:18.409506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.409437 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbb8af9-852b-4bc7-a67d-05c3fab75b7b" containerName="payload-processing" Nov 01 06:55:18.413293 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.413256 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.463565 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.463538 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx"] Nov 01 06:55:18.509945 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.509915 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"agc-cred-522cd1e62a72\"" Nov 01 06:55:18.510096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.509976 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"routing-overlay\"" Nov 01 06:55:18.510096 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.510002 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"kube-root-ca.crt\"" Nov 01 06:55:18.516413 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.516392 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"e2e-cfl-6b3232-provider-cred\"" Nov 01 06:55:18.516518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.516454 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"openshift-service-ca.crt\"" Nov 01 06:55:18.516518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.516464 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-caf74d\"" Nov 01 06:55:18.516518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.516454 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"agc-cred-dcba5d3eb853\"" Nov 01 06:55:18.516518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.516482 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"payload-processing-external-model-e2e--b10087ce-dockercfg-crgnv\"" Nov 01 06:55:18.576256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.576211 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.576388 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.576318 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.576457 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.576415 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.576513 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.576476 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjth\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.677857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.677783 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.677857 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.677830 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjth\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.678050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.677936 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.678050 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.677986 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.678487 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.678462 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.678735 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.678715 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.680808 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.680779 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:18.733969 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:18.733947 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjth\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:19.028941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:19.028856 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:19.407914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:19.407883 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx"] Nov 01 06:55:19.409251 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:55:19.409224 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e8b6dc_11e7_409b_8911_ce9cada8a0ef.slice/crio-b7de7b119c4572fa93b080cdc93861860ab1346d441c25acd60b3d8636e16043 WatchSource:0}: Error finding container b7de7b119c4572fa93b080cdc93861860ab1346d441c25acd60b3d8636e16043: Status 404 returned error can't find the container with id b7de7b119c4572fa93b080cdc93861860ab1346d441c25acd60b3d8636e16043 Nov 01 06:55:20.027544 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.027499 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" event={"ID":"88e8b6dc-11e7-409b-8911-ce9cada8a0ef","Type":"ContainerStarted","Data":"55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b"} Nov 01 06:55:20.027544 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.027550 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" event={"ID":"88e8b6dc-11e7-409b-8911-ce9cada8a0ef","Type":"ContainerStarted","Data":"b7de7b119c4572fa93b080cdc93861860ab1346d441c25acd60b3d8636e16043"} Nov 01 06:55:20.027804 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.027573 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:20.144586 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.144525 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" podStartSLOduration=2.144508213 podStartE2EDuration="2.144508213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:20.1394408 +0000 UTC m=+2735.692229994" watchObservedRunningTime="2026-11-01 06:55:20.144508213 +0000 UTC m=+2735.697297409" Nov 01 06:55:20.886526 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.886493 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx"] Nov 01 06:55:20.936014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.935984 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447"] Nov 01 06:55:20.940504 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.940486 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:20.994428 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.994406 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"agc-cred-6f862af0425e\"" Nov 01 06:55:20.998088 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:20.998066 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447"] Nov 01 06:55:21.110660 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.110626 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4j4\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.110833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.110685 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.110885 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.110856 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.110983 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.110968 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.212497 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.212413 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4j4\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.212497 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.212477 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.212744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.212544 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.212744 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.212638 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.213543 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.213519 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.213941 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.213917 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.215812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.215786 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.268915 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.268886 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4j4\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555djr447\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.554402 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.554321 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:21.930421 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:21.930392 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447"] Nov 01 06:55:21.931652 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:55:21.931616 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc313fe9e_7626_4f35_bf74_f276c23c0337.slice/crio-1f715a55bebf416a95ee4e59df9db49419583daf1d8b0eaeaa108b9381cb21e1 WatchSource:0}: Error finding container 1f715a55bebf416a95ee4e59df9db49419583daf1d8b0eaeaa108b9381cb21e1: Status 404 returned error can't find the container with id 1f715a55bebf416a95ee4e59df9db49419583daf1d8b0eaeaa108b9381cb21e1 Nov 01 06:55:22.041057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:22.041021 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" event={"ID":"c313fe9e-7626-4f35-bf74-f276c23c0337","Type":"ContainerStarted","Data":"38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8"} Nov 01 06:55:22.041201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:22.041095 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" event={"ID":"c313fe9e-7626-4f35-bf74-f276c23c0337","Type":"ContainerStarted","Data":"1f715a55bebf416a95ee4e59df9db49419583daf1d8b0eaeaa108b9381cb21e1"} Nov 01 06:55:22.041201 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:22.041160 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:22.162321 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:22.162239 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" podStartSLOduration=2.162224541 podStartE2EDuration="2.162224541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:22.160867747 +0000 UTC m=+2737.713656942" watchObservedRunningTime="2026-11-01 06:55:22.162224541 +0000 UTC m=+2737.715013755" Nov 01 06:55:27.042092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.042007 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" podUID="88e8b6dc-11e7-409b-8911-ce9cada8a0ef" containerName="payload-processing" containerID="cri-o://55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b" gracePeriod=25 Nov 01 06:55:27.359256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.359228 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:27.510518 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.510487 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume\") pod \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " Nov 01 06:55:27.510702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.510571 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials\") pod \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " Nov 01 06:55:27.510702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.510605 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth\") pod \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " Nov 01 06:55:27.510702 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.510633 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume\") pod \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\" (UID: \"88e8b6dc-11e7-409b-8911-ce9cada8a0ef\") " Nov 01 06:55:27.511007 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.510982 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume" pod "88e8b6dc-11e7-409b-8911-ce9cada8a0ef" (UID: "88e8b6dc-11e7-409b-8911-ce9cada8a0ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:55:27.511137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.511110 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume" pod "88e8b6dc-11e7-409b-8911-ce9cada8a0ef" (UID: "88e8b6dc-11e7-409b-8911-ce9cada8a0ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:55:27.512957 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.512928 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials" pod "88e8b6dc-11e7-409b-8911-ce9cada8a0ef" (UID: "88e8b6dc-11e7-409b-8911-ce9cada8a0ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:55:27.513303 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.513263 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth" pod "88e8b6dc-11e7-409b-8911-ce9cada8a0ef" (UID: "88e8b6dc-11e7-409b-8911-ce9cada8a0ef"). InnerVolumeSpecName "kube-api-access-qpjth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:55:27.612069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.612040 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:27.612069 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.612067 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjth\" (UniqueName: \"kubernetes.io/projected/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-kube-api-access-qpjth\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:27.612253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.612077 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:27.612253 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:27.612087 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e8b6dc-11e7-409b-8911-ce9cada8a0ef-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:28.079963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.079873 4914 generic.go:356] "Generic (PLEG): container finished" podID="88e8b6dc-11e7-409b-8911-ce9cada8a0ef" containerID="55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b" exitCode=0 Nov 01 06:55:28.079963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.079940 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" Nov 01 06:55:28.080506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.079956 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" event={"ID":"88e8b6dc-11e7-409b-8911-ce9cada8a0ef","Type":"ContainerDied","Data":"55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b"} Nov 01 06:55:28.080506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.080013 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx" event={"ID":"88e8b6dc-11e7-409b-8911-ce9cada8a0ef","Type":"ContainerDied","Data":"b7de7b119c4572fa93b080cdc93861860ab1346d441c25acd60b3d8636e16043"} Nov 01 06:55:28.080506 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.080036 4914 scope.go:122] "RemoveContainer" containerID="55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b" Nov 01 06:55:28.090901 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.090883 4914 scope.go:122] "RemoveContainer" containerID="55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b" Nov 01 06:55:28.091156 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:28.091137 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b\": container with ID starting with 55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b not found: ID does not exist" containerID="55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b" Nov 01 06:55:28.091205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.091164 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b"} err="failed to get container status \"55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b\": rpc error: code = NotFound desc = could not find container \"55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b\": container with ID starting with 55d657f9a99fca203a5db0f78169fb6c4089555dc51bf5bc2909be8ea1d2ff0b not found: ID does not exist" Nov 01 06:55:28.243344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.243303 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx"] Nov 01 06:55:28.293004 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.292976 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-797788kxx"] Nov 01 06:55:28.617247 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.617214 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e8b6dc-11e7-409b-8911-ce9cada8a0ef" path="/var/lib/kubelet/pods/88e8b6dc-11e7-409b-8911-ce9cada8a0ef/volumes" Nov 01 06:55:28.626750 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.626726 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-046fbf"] Nov 01 06:55:28.627337 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.627323 4914 cpu_manager.go:414] "containerMap: removing container" podUID="88e8b6dc-11e7-409b-8911-ce9cada8a0ef" containerName="payload-processing" Nov 01 06:55:28.627382 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.627339 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e8b6dc-11e7-409b-8911-ce9cada8a0ef" containerName="payload-processing" Nov 01 06:55:28.627476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.627467 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e8b6dc-11e7-409b-8911-ce9cada8a0ef" containerName="payload-processing" Nov 01 06:55:28.632205 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.632187 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:28.683260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.683240 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-046fbf"] Nov 01 06:55:28.697733 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.697715 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\"" Nov 01 06:55:28.721743 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.721715 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgjq\" (UniqueName: \"kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq\") pod \"test-curl-7686-046fbf\" (UID: \"0f3165dc-f8ad-4e7a-b344-48957d9246d0\") " pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:28.822963 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.822918 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgjq\" (UniqueName: \"kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq\") pod \"test-curl-7686-046fbf\" (UID: \"0f3165dc-f8ad-4e7a-b344-48957d9246d0\") " pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:28.878863 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.878791 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgjq\" (UniqueName: \"kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq\") pod \"test-curl-7686-046fbf\" (UID: \"0f3165dc-f8ad-4e7a-b344-48957d9246d0\") " pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:28.944053 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:28.944026 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:29.317191 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:29.317165 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-046fbf"] Nov 01 06:55:29.319158 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:55:29.319126 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f3165dc_f8ad_4e7a_b344_48957d9246d0.slice/crio-1d5bcbc13244a961c6051e52ab440a556fa58dd8f5d51e956340d5c530c2adfc WatchSource:0}: Error finding container 1d5bcbc13244a961c6051e52ab440a556fa58dd8f5d51e956340d5c530c2adfc: Status 404 returned error can't find the container with id 1d5bcbc13244a961c6051e52ab440a556fa58dd8f5d51e956340d5c530c2adfc Nov 01 06:55:30.095090 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:30.095056 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-046fbf" event={"ID":"0f3165dc-f8ad-4e7a-b344-48957d9246d0","Type":"ContainerStarted","Data":"1d5bcbc13244a961c6051e52ab440a556fa58dd8f5d51e956340d5c530c2adfc"} Nov 01 06:55:33.051227 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:33.051197 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:55:33.115883 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:33.115851 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-046fbf" event={"ID":"0f3165dc-f8ad-4e7a-b344-48957d9246d0","Type":"ContainerStarted","Data":"8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2"} Nov 01 06:55:33.274647 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:33.274594 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-046fbf" podStartSLOduration=2.396989541 podStartE2EDuration="5.274578392s" totalImagesPullingTime="2.877588851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:33.273202075 +0000 UTC m=+2748.825991269" watchObservedRunningTime="2026-11-01 06:55:33.274578392 +0000 UTC m=+2748.827367585" Nov 01 06:55:34.772349 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.772318 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-046fbf"] Nov 01 06:55:34.778267 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.778241 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-046fbf"] Nov 01 06:55:34.922095 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.922069 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-99155e"] Nov 01 06:55:34.922711 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.922697 4914 cpu_manager.go:414] "containerMap: removing container" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" containerName="test-curl-7686-046fbf" Nov 01 06:55:34.922760 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.922713 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" containerName="test-curl-7686-046fbf" Nov 01 06:55:34.922834 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.922825 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" containerName="test-curl-7686-046fbf" Nov 01 06:55:34.926395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.926379 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:34.980159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:34.980129 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-99155e"] Nov 01 06:55:35.015813 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:35.015786 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmd7\" (UniqueName: \"kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7\") pod \"test-curl-7686-99155e\" (UID: \"06a221bb-9839-4358-a049-57169437d096\") " pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:35.116854 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:35.116825 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmd7\" (UniqueName: \"kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7\") pod \"test-curl-7686-99155e\" (UID: \"06a221bb-9839-4358-a049-57169437d096\") " pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:35.128432 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:35.128401 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-046fbf" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" containerName="test-curl-7686-046fbf" containerID="cri-o://8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2" gracePeriod=2 Nov 01 06:55:35.172935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:35.172911 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmd7\" (UniqueName: \"kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7\") pod \"test-curl-7686-99155e\" (UID: \"06a221bb-9839-4358-a049-57169437d096\") " pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:35.179076 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:35.179047 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-046fbf\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:35.238594 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:35.238565 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:35.610378 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:35.610349 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-99155e"] Nov 01 06:55:35.612056 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:55:35.612022 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06a221bb_9839_4358_a049_57169437d096.slice/crio-f00142a6a36d971ba85c101ee0da05e6f4de2515c532215051fc6e50b6d28723 WatchSource:0}: Error finding container f00142a6a36d971ba85c101ee0da05e6f4de2515c532215051fc6e50b6d28723: Status 404 returned error can't find the container with id f00142a6a36d971ba85c101ee0da05e6f4de2515c532215051fc6e50b6d28723 Nov 01 06:55:36.135405 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:36.135374 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-99155e" event={"ID":"06a221bb-9839-4358-a049-57169437d096","Type":"ContainerStarted","Data":"93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53"} Nov 01 06:55:36.135405 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:36.135416 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-99155e" event={"ID":"06a221bb-9839-4358-a049-57169437d096","Type":"ContainerStarted","Data":"f00142a6a36d971ba85c101ee0da05e6f4de2515c532215051fc6e50b6d28723"} Nov 01 06:55:36.249896 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:36.249845 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-99155e" podStartSLOduration=2.249831355 podStartE2EDuration="2.249831355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:36.247023123 +0000 UTC m=+2751.799812317" watchObservedRunningTime="2026-11-01 06:55:36.249831355 +0000 UTC m=+2751.802620548" Nov 01 06:55:36.297725 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:36.297688 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-046fbf\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:37.380055 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.380020 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:37.445829 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.445771 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq\" (UniqueName: \"kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq\") pod \"0f3165dc-f8ad-4e7a-b344-48957d9246d0\" (UID: \"0f3165dc-f8ad-4e7a-b344-48957d9246d0\") " Nov 01 06:55:37.447831 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.447808 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq" pod "0f3165dc-f8ad-4e7a-b344-48957d9246d0" (UID: "0f3165dc-f8ad-4e7a-b344-48957d9246d0"). InnerVolumeSpecName "kube-api-access-bqgjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:55:37.547568 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.547543 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgjq\" (UniqueName: \"kubernetes.io/projected/0f3165dc-f8ad-4e7a-b344-48957d9246d0-kube-api-access-bqgjq\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:37.594900 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:37.594869 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:55:37.625385 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.625348 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-99155e"] Nov 01 06:55:37.628370 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.628343 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-99155e"] Nov 01 06:55:37.772810 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.772734 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7686-13cf87"] Nov 01 06:55:37.773420 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.773405 4914 cpu_manager.go:414] "containerMap: removing container" podUID="06a221bb-9839-4358-a049-57169437d096" containerName="test-curl-7686-99155e" Nov 01 06:55:37.773472 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.773422 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a221bb-9839-4358-a049-57169437d096" containerName="test-curl-7686-99155e" Nov 01 06:55:37.773525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.773515 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a221bb-9839-4358-a049-57169437d096" containerName="test-curl-7686-99155e" Nov 01 06:55:37.777197 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.777178 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:37.829598 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.829573 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-13cf87"] Nov 01 06:55:37.850704 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.850674 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwnd\" (UniqueName: \"kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd\") pod \"test-curl-7686-13cf87\" (UID: \"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e\") " pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:37.952198 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:37.952163 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwnd\" (UniqueName: \"kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd\") pod \"test-curl-7686-13cf87\" (UID: \"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e\") " pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:38.008193 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.008160 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwnd\" (UniqueName: \"kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd\") pod \"test-curl-7686-13cf87\" (UID: \"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e\") " pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:38.088765 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.088695 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:38.159306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.155703 4914 generic.go:356] "Generic (PLEG): container finished" podID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" containerID="8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2" exitCode=137 Nov 01 06:55:38.159306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.155926 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-99155e" podUID="06a221bb-9839-4358-a049-57169437d096" containerName="test-curl-7686-99155e" containerID="cri-o://93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53" gracePeriod=2 Nov 01 06:55:38.159306 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.156331 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:38.162151 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.162127 4914 scope.go:122] "RemoveContainer" containerID="8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2" Nov 01 06:55:38.173160 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.173135 4914 scope.go:122] "RemoveContainer" containerID="8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2" Nov 01 06:55:38.173528 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:38.173506 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2\": container with ID starting with 8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2 not found: ID does not exist" containerID="8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2" Nov 01 06:55:38.173638 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.173535 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2"} err="failed to get container status \"8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2\": rpc error: code = NotFound desc = could not find container \"8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2\": container with ID starting with 8f9a80922d36a3f0209155c13c04ac9d3dffcc0cf2a89ebcd4928c3d443114e2 not found: ID does not exist" Nov 01 06:55:38.212612 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:38.212574 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-99155e\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="06a221bb-9839-4358-a049-57169437d096" pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:38.263181 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:38.263146 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-99155e\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="06a221bb-9839-4358-a049-57169437d096" pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:38.313681 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:38.313651 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-046fbf\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" pod="opendatahub/test-curl-7686-046fbf" Nov 01 06:55:38.463189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.463147 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7686-13cf87"] Nov 01 06:55:38.464648 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:55:38.464621 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod218119ce_9ee9_4c45_8fbe_eae6b9cf2f5e.slice/crio-b3d61bad9de8eb3ee2c87f3ebe8521f9693196843f53deaa02f877aab52e2d94 WatchSource:0}: Error finding container b3d61bad9de8eb3ee2c87f3ebe8521f9693196843f53deaa02f877aab52e2d94: Status 404 returned error can't find the container with id b3d61bad9de8eb3ee2c87f3ebe8521f9693196843f53deaa02f877aab52e2d94 Nov 01 06:55:38.617797 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:38.617762 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3165dc-f8ad-4e7a-b344-48957d9246d0" path="/var/lib/kubelet/pods/0f3165dc-f8ad-4e7a-b344-48957d9246d0/volumes" Nov 01 06:55:39.168729 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:39.168697 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-13cf87" event={"ID":"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e","Type":"ContainerStarted","Data":"483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971"} Nov 01 06:55:39.168914 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:39.168739 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7686-13cf87" event={"ID":"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e","Type":"ContainerStarted","Data":"b3d61bad9de8eb3ee2c87f3ebe8521f9693196843f53deaa02f877aab52e2d94"} Nov 01 06:55:39.282315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:39.282247 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7686-13cf87" podStartSLOduration=2.282233695 podStartE2EDuration="2.282233695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:55:39.279465505 +0000 UTC m=+2754.832254699" watchObservedRunningTime="2026-11-01 06:55:39.282233695 +0000 UTC m=+2754.835022889" Nov 01 06:55:40.397773 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.397746 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:40.479351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.479262 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7\" (UniqueName: \"kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7\") pod \"06a221bb-9839-4358-a049-57169437d096\" (UID: \"06a221bb-9839-4358-a049-57169437d096\") " Nov 01 06:55:40.481535 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.481512 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7" pod "06a221bb-9839-4358-a049-57169437d096" (UID: "06a221bb-9839-4358-a049-57169437d096"). InnerVolumeSpecName "kube-api-access-8zmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:55:40.581092 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.581060 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmd7\" (UniqueName: \"kubernetes.io/projected/06a221bb-9839-4358-a049-57169437d096-kube-api-access-8zmd7\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:40.616650 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.616620 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a221bb-9839-4358-a049-57169437d096" path="/var/lib/kubelet/pods/06a221bb-9839-4358-a049-57169437d096/volumes" Nov 01 06:55:40.722962 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.722925 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7686-13cf87"] Nov 01 06:55:40.726052 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:40.726026 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7686-13cf87"] Nov 01 06:55:41.183986 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:41.183955 4914 generic.go:356] "Generic (PLEG): container finished" podID="06a221bb-9839-4358-a049-57169437d096" containerID="93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53" exitCode=137 Nov 01 06:55:41.184137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:41.184013 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-99155e" Nov 01 06:55:41.184137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:41.184050 4914 scope.go:122] "RemoveContainer" containerID="93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53" Nov 01 06:55:41.184202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:41.184154 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7686-13cf87" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" containerName="test-curl-7686-13cf87" containerID="cri-o://483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971" gracePeriod=2 Nov 01 06:55:41.194159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:41.194136 4914 scope.go:122] "RemoveContainer" containerID="93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53" Nov 01 06:55:41.194477 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:41.194446 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53\": container with ID starting with 93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53 not found: ID does not exist" containerID="93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53" Nov 01 06:55:41.194541 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:41.194477 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53"} err="failed to get container status \"93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53\": rpc error: code = NotFound desc = could not find container \"93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53\": container with ID starting with 93c29ec685b75bfdbe4484899fc7a287fe1eeb43eeddce577cdc361b892cae53 not found: ID does not exist" Nov 01 06:55:41.235071 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:41.235032 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-13cf87\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:41.285781 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:41.285747 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-13cf87\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:43.437905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:43.437843 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:43.512948 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:43.512923 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd\" (UniqueName: \"kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd\") pod \"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e\" (UID: \"218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e\") " Nov 01 06:55:43.515024 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:43.514999 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd" pod "218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" (UID: "218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e"). InnerVolumeSpecName "kube-api-access-xrwnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:55:43.614539 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:43.614510 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwnd\" (UniqueName: \"kubernetes.io/projected/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e-kube-api-access-xrwnd\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:55:44.204545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:44.204509 4914 generic.go:356] "Generic (PLEG): container finished" podID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" containerID="483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971" exitCode=137 Nov 01 06:55:44.204723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:44.204552 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:44.204723 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:44.204575 4914 scope.go:122] "RemoveContainer" containerID="483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971" Nov 01 06:55:44.214336 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:44.214318 4914 scope.go:122] "RemoveContainer" containerID="483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971" Nov 01 06:55:44.214590 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:44.214571 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971\": container with ID starting with 483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971 not found: ID does not exist" containerID="483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971" Nov 01 06:55:44.214643 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:44.214598 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971"} err="failed to get container status \"483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971\": rpc error: code = NotFound desc = could not find container \"483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971\": container with ID starting with 483f54c0f8c7af27923c297eb6d39d1345fb33bd63c1021eb2db94b3ed4e2971 not found: ID does not exist" Nov 01 06:55:44.263966 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:44.263927 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-13cf87\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:55:44.617027 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:55:44.616996 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" path="/var/lib/kubelet/pods/218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e/volumes" Nov 01 06:55:44.663085 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:55:44.663054 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7686-13cf87\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" pod="opendatahub/test-curl-7686-13cf87" Nov 01 06:56:07.593257 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:56:07.593222 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:56:19.019109 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.019077 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh"] Nov 01 06:56:19.020589 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.020563 4914 cpu_manager.go:414] "containerMap: removing container" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" containerName="test-curl-7686-13cf87" Nov 01 06:56:19.020737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.020723 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" containerName="test-curl-7686-13cf87" Nov 01 06:56:19.020988 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.020975 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="218119ce-9ee9-4c45-8fbe-eae6b9cf2f5e" containerName="test-curl-7686-13cf87" Nov 01 06:56:19.039776 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.039746 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.081057 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.081032 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh"] Nov 01 06:56:19.089119 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089090 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089140 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089180 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089220 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089214 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089388 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089371 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089426 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089417 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089459 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089446 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089496 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089482 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt9kr\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.089532 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.089498 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.127288 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.127247 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-28c598-openshift-default-dockercfg-z7ljk\"" Nov 01 06:56:19.191098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191060 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191102 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191163 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191251 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191334 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191382 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9kr\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191409 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191407 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191502 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191534 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191647 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.191713 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191658 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.192093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191787 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.192093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.191864 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.192244 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.192223 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.193596 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.193573 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.193917 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.193899 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.263570 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.263493 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.267988 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.267954 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9kr\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr\") pod \"e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.360968 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.360524 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:19.858553 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.858523 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh"] Nov 01 06:56:19.927331 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:56:19.927296 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95ad465_57bc_4905_861a_cefcd9a6b717.slice/crio-a86924c97a03939d6addc45b7c89ae1221cf790974be4159cf4ead8380d583c1 WatchSource:0}: Error finding container a86924c97a03939d6addc45b7c89ae1221cf790974be4159cf4ead8380d583c1: Status 404 returned error can't find the container with id a86924c97a03939d6addc45b7c89ae1221cf790974be4159cf4ead8380d583c1 Nov 01 06:56:19.929905 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.929869 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:56:19.929995 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.929942 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:56:19.929995 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:19.929971 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 06:56:20.113911 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:20.113836 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt"] Nov 01 06:56:20.434372 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:20.432855 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" event={"ID":"c95ad465-57bc-4905-861a-cefcd9a6b717","Type":"ContainerStarted","Data":"bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06"} Nov 01 06:56:20.434372 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:20.432916 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" event={"ID":"c95ad465-57bc-4905-861a-cefcd9a6b717","Type":"ContainerStarted","Data":"a86924c97a03939d6addc45b7c89ae1221cf790974be4159cf4ead8380d583c1"} Nov 01 06:56:20.568346 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:20.561140 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podStartSLOduration=2.561120807 podStartE2EDuration="2.561120807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:20.559895772 +0000 UTC m=+2796.112684968" watchObservedRunningTime="2026-11-01 06:56:20.561120807 +0000 UTC m=+2796.113910002" Nov 01 06:56:21.366366 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:21.366333 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:21.370442 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:21.370412 4914 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:56:21.370703 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:21.370675 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:56:22.361725 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:22.361687 4914 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:56:22.362020 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:22.361992 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:56:23.367300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:23.361613 4914 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:56:23.367300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:23.361683 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:56:24.368300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:24.367685 4914 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:56:24.368300 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:24.367761 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:56:25.114642 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.114587 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" podUID="52d5d3cc-966f-4d8a-8409-ef15d0697778" containerName="payload-processing" containerID="cri-o://98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918" gracePeriod=25 Nov 01 06:56:25.369351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.368651 4914 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:56:25.369351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.368714 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:56:25.409313 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.409266 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:56:25.489180 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.489146 4914 generic.go:356] "Generic (PLEG): container finished" podID="52d5d3cc-966f-4d8a-8409-ef15d0697778" containerID="98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918" exitCode=0 Nov 01 06:56:25.489359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.489183 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" event={"ID":"52d5d3cc-966f-4d8a-8409-ef15d0697778","Type":"ContainerDied","Data":"98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918"} Nov 01 06:56:25.489359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.489241 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" event={"ID":"52d5d3cc-966f-4d8a-8409-ef15d0697778","Type":"ContainerDied","Data":"a15d3bbf2b9134f9a4f1aaefc748b30a0eabd03c4144353da4bab5e76a41e9a2"} Nov 01 06:56:25.489359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.489263 4914 scope.go:122] "RemoveContainer" containerID="98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918" Nov 01 06:56:25.489359 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.489191 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt" Nov 01 06:56:25.499059 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.499042 4914 scope.go:122] "RemoveContainer" containerID="98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918" Nov 01 06:56:25.499360 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:56:25.499338 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918\": container with ID starting with 98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918 not found: ID does not exist" containerID="98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918" Nov 01 06:56:25.499434 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.499371 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918"} err="failed to get container status \"98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918\": rpc error: code = NotFound desc = could not find container \"98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918\": container with ID starting with 98130e922bfa982df42d79f3c6e0f7926c577bebc7f408053812cdd40f580918 not found: ID does not exist" Nov 01 06:56:25.518693 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.518666 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume\") pod \"52d5d3cc-966f-4d8a-8409-ef15d0697778\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " Nov 01 06:56:25.518784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.518722 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr\" (UniqueName: \"kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr\") pod \"52d5d3cc-966f-4d8a-8409-ef15d0697778\" (UID: \"52d5d3cc-966f-4d8a-8409-ef15d0697778\") " Nov 01 06:56:25.518994 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.518975 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume" pod "52d5d3cc-966f-4d8a-8409-ef15d0697778" (UID: "52d5d3cc-966f-4d8a-8409-ef15d0697778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:56:25.520795 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.520779 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr" pod "52d5d3cc-966f-4d8a-8409-ef15d0697778" (UID: "52d5d3cc-966f-4d8a-8409-ef15d0697778"). InnerVolumeSpecName "kube-api-access-sxdrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:56:25.620338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.620242 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdrr\" (UniqueName: \"kubernetes.io/projected/52d5d3cc-966f-4d8a-8409-ef15d0697778-kube-api-access-sxdrr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:25.620338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.620284 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d5d3cc-966f-4d8a-8409-ef15d0697778-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:56:25.933957 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.933880 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt"] Nov 01 06:56:25.937049 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:25.937025 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a3e841-5d4f79784f-pgmlt"] Nov 01 06:56:26.361607 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:26.361568 4914 patch_prober.go:29] interesting pod/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 01 06:56:26.361794 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:26.361620 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 01 06:56:26.620073 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:26.620002 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d5d3cc-966f-4d8a-8409-ef15d0697778" path="/var/lib/kubelet/pods/52d5d3cc-966f-4d8a-8409-ef15d0697778/volumes" Nov 01 06:56:27.366871 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:27.366837 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:27.369125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:27.367191 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:27.369125 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:27.367843 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 06:56:28.926935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.926893 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x"] Nov 01 06:56:28.927593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.927576 4914 cpu_manager.go:414] "containerMap: removing container" podUID="52d5d3cc-966f-4d8a-8409-ef15d0697778" containerName="payload-processing" Nov 01 06:56:28.927593 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.927594 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d5d3cc-966f-4d8a-8409-ef15d0697778" containerName="payload-processing" Nov 01 06:56:28.927761 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.927750 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d5d3cc-966f-4d8a-8409-ef15d0697778" containerName="payload-processing" Nov 01 06:56:28.931343 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.931324 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:28.965912 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.965883 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqgz9\" (UniqueName: \"kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9\") pod \"payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:28.966006 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.965939 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:28.987315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:28.987259 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x"] Nov 01 06:56:29.043997 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.042500 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-3db672\"" Nov 01 06:56:29.045116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.044427 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-3db672-dockercfg-85lhg\"" Nov 01 06:56:29.067344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.066712 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:29.067344 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.066930 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqgz9\" (UniqueName: \"kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9\") pod \"payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:29.068935 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.068903 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:29.123657 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.123625 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqgz9\" (UniqueName: \"kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9\") pod \"payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:29.244813 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.244727 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:29.618603 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:29.618573 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x"] Nov 01 06:56:29.619930 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:56:29.619909 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f50f78f_ecdb_4a95_8746_7aa30b9c4044.slice/crio-1060af341b13e4e4667fe54b32ddd929fba206144ad8dd1fff83b479cfad5802 WatchSource:0}: Error finding container 1060af341b13e4e4667fe54b32ddd929fba206144ad8dd1fff83b479cfad5802: Status 404 returned error can't find the container with id 1060af341b13e4e4667fe54b32ddd929fba206144ad8dd1fff83b479cfad5802 Nov 01 06:56:30.526430 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:30.526397 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" event={"ID":"0f50f78f-ecdb-4a95-8746-7aa30b9c4044","Type":"ContainerStarted","Data":"e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484"} Nov 01 06:56:30.526821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:30.526444 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" event={"ID":"0f50f78f-ecdb-4a95-8746-7aa30b9c4044","Type":"ContainerStarted","Data":"1060af341b13e4e4667fe54b32ddd929fba206144ad8dd1fff83b479cfad5802"} Nov 01 06:56:30.526821 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:30.526620 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:37.593679 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:56:37.593644 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:56:41.535641 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:41.535614 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 06:56:41.648663 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:41.648612 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" podStartSLOduration=13.648600577 podStartE2EDuration="13.648600577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:30.646347033 +0000 UTC m=+2806.199136231" watchObservedRunningTime="2026-11-01 06:56:41.648600577 +0000 UTC m=+2817.201389770" Nov 01 06:56:49.325296 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.325237 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp"] Nov 01 06:56:49.330880 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.330858 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.386256 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.386223 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp"] Nov 01 06:56:49.414611 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.414568 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.415137 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.415108 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2dq\" (UniqueName: \"kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.445795 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.445757 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c831a4c0\"" Nov 01 06:56:49.446170 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.446150 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c831a4c0-dockercfg-6pk2x\"" Nov 01 06:56:49.516351 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.516313 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2dq\" (UniqueName: \"kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.516546 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.516525 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.517398 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.517373 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.573178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.573142 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2dq\" (UniqueName: \"kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq\") pod \"payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:49.646302 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:49.644579 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:50.034451 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:50.034427 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp"] Nov 01 06:56:50.036424 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:56:50.036394 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3854de33_4b34_44d3_a618_4f6630aad360.slice/crio-fbbdab284c1f275e34fb05260f383e1e7d416c091bde51ff7f30e4f8e6dc842c WatchSource:0}: Error finding container fbbdab284c1f275e34fb05260f383e1e7d416c091bde51ff7f30e4f8e6dc842c: Status 404 returned error can't find the container with id fbbdab284c1f275e34fb05260f383e1e7d416c091bde51ff7f30e4f8e6dc842c Nov 01 06:56:50.661334 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:50.661303 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" event={"ID":"3854de33-4b34-44d3-a618-4f6630aad360","Type":"ContainerStarted","Data":"715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041"} Nov 01 06:56:50.661762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:50.661353 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" event={"ID":"3854de33-4b34-44d3-a618-4f6630aad360","Type":"ContainerStarted","Data":"fbbdab284c1f275e34fb05260f383e1e7d416c091bde51ff7f30e4f8e6dc842c"} Nov 01 06:56:50.661762 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:50.661400 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:56:50.775517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:50.775458 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" podStartSLOduration=1.775441379 podStartE2EDuration="1.775441379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:56:50.773767891 +0000 UTC m=+2826.326557085" watchObservedRunningTime="2026-11-01 06:56:50.775441379 +0000 UTC m=+2826.328230574" Nov 01 06:56:59.259678 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.259464 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc"] Nov 01 06:56:59.267259 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.267230 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.309737 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.309704 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc"] Nov 01 06:56:59.355200 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.355163 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0cb4ef\"" Nov 01 06:56:59.355383 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.355210 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0cb4ef-dockercfg-qxjg2\"" Nov 01 06:56:59.363839 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.363815 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.363977 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.363954 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxrmn\" (UniqueName: \"kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn\") pod \"payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.465578 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.465547 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxrmn\" (UniqueName: \"kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn\") pod \"payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.465748 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.465727 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.466486 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.466465 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.522041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.521974 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxrmn\" (UniqueName: \"kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn\") pod \"payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.582787 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.582762 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:56:59.954170 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:56:59.954141 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc"] Nov 01 06:56:59.955811 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:56:59.955784 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b0c163_5063_4404_8d8d_8f73d223d0ca.slice/crio-a343f0ea17125b31a17dbb002bf08a3f6b8879ce27f9dbfe746f4956c34efff0 WatchSource:0}: Error finding container a343f0ea17125b31a17dbb002bf08a3f6b8879ce27f9dbfe746f4956c34efff0: Status 404 returned error can't find the container with id a343f0ea17125b31a17dbb002bf08a3f6b8879ce27f9dbfe746f4956c34efff0 Nov 01 06:57:00.725515 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:00.725484 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" event={"ID":"61b0c163-5063-4404-8d8d-8f73d223d0ca","Type":"ContainerStarted","Data":"60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595"} Nov 01 06:57:00.725900 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:00.725533 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" event={"ID":"61b0c163-5063-4404-8d8d-8f73d223d0ca","Type":"ContainerStarted","Data":"a343f0ea17125b31a17dbb002bf08a3f6b8879ce27f9dbfe746f4956c34efff0"} Nov 01 06:57:00.725900 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:00.725675 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:57:00.839093 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:00.839021 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" podStartSLOduration=1.839003971 podStartE2EDuration="1.839003971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:00.837566871 +0000 UTC m=+2836.390356066" watchObservedRunningTime="2026-11-01 06:57:00.839003971 +0000 UTC m=+2836.391793166" Nov 01 06:57:01.671386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:01.671357 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 06:57:07.593147 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:57:07.593114 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:57:11.733812 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:11.733779 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 06:57:37.095998 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.095958 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp"] Nov 01 06:57:37.101679 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.101656 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.153463 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.153436 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.153608 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.153578 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxbf\" (UniqueName: \"kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.153672 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.153657 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.153735 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.153689 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.154190 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.154168 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3db672-serving-cert\"" Nov 01 06:57:37.206920 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.206892 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp"] Nov 01 06:57:37.254825 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.254797 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.254999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.254914 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxbf\" (UniqueName: \"kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.254999 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.254973 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.255122 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:57:37.255004 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-serving-cert: secret "maas-api-e2e-shared-a-w2-3db672-serving-cert" not found Nov 01 06:57:37.255122 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.255013 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.255122 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:57:37.255086 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls podName:9d434462-1888-45d1-a387-e0eea3df84b6 nodeName:}" failed. No retries permitted until 2026-11-01 06:57:37.755063848 +0000 UTC m=+2873.307853024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls") pod "maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" (UID: "9d434462-1888-45d1-a387-e0eea3df84b6") : secret "maas-api-e2e-shared-a-w2-3db672-serving-cert" not found Nov 01 06:57:37.255665 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.255641 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.257664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.257623 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.311783 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.311756 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxbf\" (UniqueName: \"kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.595265 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:57:37.595230 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:57:37.775388 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.775311 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:37.778045 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:37.778022 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:38.015766 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:38.015733 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:38.389411 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:38.389384 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp"] Nov 01 06:57:38.390904 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:57:38.390880 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d434462_1888_45d1_a387_e0eea3df84b6.slice/crio-66c16685c621fb566da862f4f5e7e916affaa13e77f21f278a95a29716f38179 WatchSource:0}: Error finding container 66c16685c621fb566da862f4f5e7e916affaa13e77f21f278a95a29716f38179: Status 404 returned error can't find the container with id 66c16685c621fb566da862f4f5e7e916affaa13e77f21f278a95a29716f38179 Nov 01 06:57:38.392404 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:38.392383 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 06:57:38.972525 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:38.972491 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" event={"ID":"9d434462-1888-45d1-a387-e0eea3df84b6","Type":"ContainerStarted","Data":"66c16685c621fb566da862f4f5e7e916affaa13e77f21f278a95a29716f38179"} Nov 01 06:57:39.980376 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:39.980340 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" event={"ID":"9d434462-1888-45d1-a387-e0eea3df84b6","Type":"ContainerStarted","Data":"df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3"} Nov 01 06:57:39.980779 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:39.980398 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:57:40.105654 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:40.105598 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" podStartSLOduration=2.191573096 podStartE2EDuration="3.105583479s" totalImagesPullingTime="914.010383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:57:40.101985466 +0000 UTC m=+2875.654774659" watchObservedRunningTime="2026-11-01 06:57:40.105583479 +0000 UTC m=+2875.658372672" Nov 01 06:57:45.992626 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:57:45.992591 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 06:58:07.595811 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:58:07.595766 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:58:15.090008 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.089975 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f"] Nov 01 06:58:15.094261 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.094242 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.146546 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.146515 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0cb4ef-serving-cert\"" Nov 01 06:58:15.195013 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.194980 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79f2h\" (UniqueName: \"kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.195159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.195042 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.195159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.195125 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.195233 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.195167 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.199385 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.199362 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f"] Nov 01 06:58:15.295782 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.295748 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.295950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.295795 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.295950 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.295841 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.296067 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:58:15.295957 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-serving-cert: secret "maas-api-e2e-explicit-0cb4ef-serving-cert" not found Nov 01 06:58:15.296067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.296005 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79f2h\" (UniqueName: \"kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.296067 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:58:15.296015 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls podName:c9873d7a-fc07-4dfa-8978-f560226a63c0 nodeName:}" failed. No retries permitted until 2026-11-01 06:58:15.795999581 +0000 UTC m=+2911.348788756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls") pod "maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" (UID: "c9873d7a-fc07-4dfa-8978-f560226a63c0") : secret "maas-api-e2e-explicit-0cb4ef-serving-cert" not found Nov 01 06:58:15.296644 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.296618 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.298353 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.298334 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.371726 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.371692 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79f2h\" (UniqueName: \"kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.801648 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.801561 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:15.804159 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:15.804129 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\") pod \"maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:16.007305 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:16.007253 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:16.381664 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:16.381634 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f"] Nov 01 06:58:16.382828 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:58:16.382802 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9873d7a_fc07_4dfa_8978_f560226a63c0.slice/crio-ed914b784e3fdeefbe5e9912213bee0e865b7061e440478008d70913db9fbd24 WatchSource:0}: Error finding container ed914b784e3fdeefbe5e9912213bee0e865b7061e440478008d70913db9fbd24: Status 404 returned error can't find the container with id ed914b784e3fdeefbe5e9912213bee0e865b7061e440478008d70913db9fbd24 Nov 01 06:58:17.208178 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:17.208150 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" event={"ID":"c9873d7a-fc07-4dfa-8978-f560226a63c0","Type":"ContainerStarted","Data":"ed914b784e3fdeefbe5e9912213bee0e865b7061e440478008d70913db9fbd24"} Nov 01 06:58:18.216556 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:18.216512 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" event={"ID":"c9873d7a-fc07-4dfa-8978-f560226a63c0","Type":"ContainerStarted","Data":"3a471feffb66dd069c3c8386360c025d418980d30dfab314c471a1bda14b1aed"} Nov 01 06:58:18.217014 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:18.216728 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:18.331532 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:18.331473 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" podStartSLOduration=2.511330901 podStartE2EDuration="3.331455319s" totalImagesPullingTime="820.124418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:58:18.329544957 +0000 UTC m=+2913.882334152" watchObservedRunningTime="2026-11-01 06:58:18.331455319 +0000 UTC m=+2913.884244516" Nov 01 06:58:24.229046 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:58:24.229016 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 06:58:37.593704 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:58:37.593672 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:59:07.592731 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:59:07.592695 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:59:21.374401 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.374339 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76"] Nov 01 06:59:21.379338 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.379312 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.414066 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.414025 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sjg\" (UniqueName: \"kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg\") pod \"payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.414249 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.414167 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.431041 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.431000 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76"] Nov 01 06:59:21.450540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.450503 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c52bfa-dockercfg-k58w4\"" Nov 01 06:59:21.450540 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.450516 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c52bfa\"" Nov 01 06:59:21.515796 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.515764 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sjg\" (UniqueName: \"kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg\") pod \"payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.515956 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.515852 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.516563 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.516544 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.572403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.572375 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sjg\" (UniqueName: \"kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg\") pod \"payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:21.732350 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:21.725541 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:22.122620 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:59:22.122591 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d5cda5_cac4_4296_bb88_d74ace000331.slice/crio-97a6c806d7a3f771585fc8d90a56181c705c82cc25902e6ab84b5cf4dea617c1 WatchSource:0}: Error finding container 97a6c806d7a3f771585fc8d90a56181c705c82cc25902e6ab84b5cf4dea617c1: Status 404 returned error can't find the container with id 97a6c806d7a3f771585fc8d90a56181c705c82cc25902e6ab84b5cf4dea617c1 Nov 01 06:59:22.123545 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:22.123516 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76"] Nov 01 06:59:22.654970 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:22.654940 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" event={"ID":"89d5cda5-cac4-4296-bb88-d74ace000331","Type":"ContainerStarted","Data":"f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0"} Nov 01 06:59:22.655386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:22.654989 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" event={"ID":"89d5cda5-cac4-4296-bb88-d74ace000331","Type":"ContainerStarted","Data":"97a6c806d7a3f771585fc8d90a56181c705c82cc25902e6ab84b5cf4dea617c1"} Nov 01 06:59:22.655386 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:22.655061 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:22.768417 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:22.768373 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" podStartSLOduration=1.76835961 podStartE2EDuration="1.76835961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:22.766235351 +0000 UTC m=+2978.319024544" watchObservedRunningTime="2026-11-01 06:59:22.76835961 +0000 UTC m=+2978.321148800" Nov 01 06:59:25.414310 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.414262 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b"] Nov 01 06:59:25.418784 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.418764 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.472228 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.472200 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.472403 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.472261 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.472476 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.472403 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8pmw\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.472567 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.472486 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.474685 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.474661 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b"] Nov 01 06:59:25.520395 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.520368 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-caf74d\"/\"agc-cred-ea026b0e6c64\"" Nov 01 06:59:25.574010 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.573974 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.574132 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.574046 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.574197 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.574154 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8pmw\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.574260 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.574247 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.574755 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.574736 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.574856 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.574836 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.577018 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.577002 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.630618 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.630598 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8pmw\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:25.732517 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:25.732463 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:26.109580 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:26.109555 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b"] Nov 01 06:59:26.110581 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:59:26.110559 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00a77b6_a06b_42a7_b825_c6cb64b40551.slice/crio-62da585e37cba30022885d4adb5b318b319b5d72d774afa274148f1f87fdce68 WatchSource:0}: Error finding container 62da585e37cba30022885d4adb5b318b319b5d72d774afa274148f1f87fdce68: Status 404 returned error can't find the container with id 62da585e37cba30022885d4adb5b318b319b5d72d774afa274148f1f87fdce68 Nov 01 06:59:26.679673 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:26.679642 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" event={"ID":"c00a77b6-a06b-42a7-b825-c6cb64b40551","Type":"ContainerStarted","Data":"255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f"} Nov 01 06:59:26.680067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:26.679686 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" event={"ID":"c00a77b6-a06b-42a7-b825-c6cb64b40551","Type":"ContainerStarted","Data":"62da585e37cba30022885d4adb5b318b319b5d72d774afa274148f1f87fdce68"} Nov 01 06:59:26.680067 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:26.679775 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:26.796913 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:26.796862 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" podStartSLOduration=1.796847816 podStartE2EDuration="1.796847816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 06:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 06:59:26.792697424 +0000 UTC m=+2982.345486619" watchObservedRunningTime="2026-11-01 06:59:26.796847816 +0000 UTC m=+2982.349637009" Nov 01 06:59:33.664034 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:33.664006 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 06:59:37.593056 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:59:37.593017 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 06:59:37.689510 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:37.689485 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 06:59:37.856677 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:37.856643 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447"] Nov 01 06:59:42.858042 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:42.857996 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" podUID="c313fe9e-7626-4f35-bf74-f276c23c0337" containerName="payload-processing" containerID="cri-o://38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8" gracePeriod=25 Nov 01 06:59:43.122152 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.122094 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:59:43.223199 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223169 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume\") pod \"c313fe9e-7626-4f35-bf74-f276c23c0337\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " Nov 01 06:59:43.223391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223214 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume\") pod \"c313fe9e-7626-4f35-bf74-f276c23c0337\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " Nov 01 06:59:43.223391 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223256 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials\") pod \"c313fe9e-7626-4f35-bf74-f276c23c0337\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " Nov 01 06:59:43.223527 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223496 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4\") pod \"c313fe9e-7626-4f35-bf74-f276c23c0337\" (UID: \"c313fe9e-7626-4f35-bf74-f276c23c0337\") " Nov 01 06:59:43.223590 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223571 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume" pod "c313fe9e-7626-4f35-bf74-f276c23c0337" (UID: "c313fe9e-7626-4f35-bf74-f276c23c0337"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:59:43.223646 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223623 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume" pod "c313fe9e-7626-4f35-bf74-f276c23c0337" (UID: "c313fe9e-7626-4f35-bf74-f276c23c0337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 06:59:43.223910 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223891 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:59:43.224012 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.223915 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c313fe9e-7626-4f35-bf74-f276c23c0337-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:59:43.225565 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.225534 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials" pod "c313fe9e-7626-4f35-bf74-f276c23c0337" (UID: "c313fe9e-7626-4f35-bf74-f276c23c0337"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:59:43.225662 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.225581 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4" pod "c313fe9e-7626-4f35-bf74-f276c23c0337" (UID: "c313fe9e-7626-4f35-bf74-f276c23c0337"). InnerVolumeSpecName "kube-api-access-7b4j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 06:59:43.325202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.325174 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4j4\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-kube-api-access-7b4j4\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:59:43.325202 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.325200 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c313fe9e-7626-4f35-bf74-f276c23c0337-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 06:59:43.799833 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.799798 4914 generic.go:356] "Generic (PLEG): container finished" podID="c313fe9e-7626-4f35-bf74-f276c23c0337" containerID="38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8" exitCode=0 Nov 01 06:59:43.800002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.799844 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" Nov 01 06:59:43.800002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.799878 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" event={"ID":"c313fe9e-7626-4f35-bf74-f276c23c0337","Type":"ContainerDied","Data":"38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8"} Nov 01 06:59:43.800002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.799947 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" event={"ID":"c313fe9e-7626-4f35-bf74-f276c23c0337","Type":"ContainerDied","Data":"1f715a55bebf416a95ee4e59df9db49419583daf1d8b0eaeaa108b9381cb21e1"} Nov 01 06:59:43.800002 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.799977 4914 scope.go:122] "RemoveContainer" containerID="38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8" Nov 01 06:59:43.810315 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.810299 4914 scope.go:122] "RemoveContainer" containerID="38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8" Nov 01 06:59:43.810563 ip-10-0-130-54 kubenswrapper[4914]: E1101 06:59:43.810543 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8\": container with ID starting with 38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8 not found: ID does not exist" containerID="38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8" Nov 01 06:59:43.810610 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.810571 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8"} err="failed to get container status \"38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8\": rpc error: code = NotFound desc = could not find container \"38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8\": container with ID starting with 38558278979c10a52b94d06301b4390819430ced02f44d84d11c1d3b64b172c8 not found: ID does not exist" Nov 01 06:59:43.963046 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:43.963020 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447"] Nov 01 06:59:44.013098 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:44.013073 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447"] Nov 01 06:59:44.048189 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:44.048154 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555djr447" podUID="c313fe9e-7626-4f35-bf74-f276c23c0337" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.146:50052\" within 1s: context deadline exceeded" Nov 01 06:59:44.048307 ip-10-0-130-54 kubenswrapper[4914]: W1101 06:59:44.048211 4914 logging.go:55] [core] [Channel #9878 SubChannel #9879]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.146:50052", ServerName: "10.132.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.146:50052: operation was canceled" Nov 01 06:59:44.618480 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:44.618449 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c313fe9e-7626-4f35-bf74-f276c23c0337" path="/var/lib/kubelet/pods/c313fe9e-7626-4f35-bf74-f276c23c0337/volumes" Nov 01 06:59:44.979447 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:44.979372 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 06:59:44.986116 ip-10-0-130-54 kubenswrapper[4914]: I1101 06:59:44.986093 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:00:07.593301 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:00:07.593200 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:00:37.592523 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:00:37.592480 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:00:43.161494 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.161461 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf"] Nov 01 07:00:43.162215 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.162198 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c313fe9e-7626-4f35-bf74-f276c23c0337" containerName="payload-processing" Nov 01 07:00:43.162293 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.162216 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c313fe9e-7626-4f35-bf74-f276c23c0337" containerName="payload-processing" Nov 01 07:00:43.162368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.162358 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c313fe9e-7626-4f35-bf74-f276c23c0337" containerName="payload-processing" Nov 01 07:00:43.166089 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.166067 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.217873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.217846 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c52bfa-serving-cert\"" Nov 01 07:00:43.271198 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.271150 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf"] Nov 01 07:00:43.288045 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.288019 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.288215 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.288190 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzlr\" (UniqueName: \"kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.288366 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.288337 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.288424 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.288395 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.389577 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.389544 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.389735 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.389643 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzlr\" (UniqueName: \"kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.389735 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.389688 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.389735 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:00:43.389692 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-serving-cert: secret "maas-api-e2e-shared-b-w2-c52bfa-serving-cert" not found Nov 01 07:00:43.389735 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.389723 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.389925 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:00:43.389761 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls podName:3328298e-18f1-47b9-8577-3a3344f35155 nodeName:}" failed. No retries permitted until 2026-11-01 07:00:43.889741644 +0000 UTC m=+3059.442530817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" (UID: "3328298e-18f1-47b9-8577-3a3344f35155") : secret "maas-api-e2e-shared-b-w2-c52bfa-serving-cert" not found Nov 01 07:00:43.390325 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.390294 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.392391 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.392368 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.455136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.455047 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzlr\" (UniqueName: \"kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.902654 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.902618 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:43.905255 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:43.905230 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:44.079971 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:44.079946 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:44.454236 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:44.454209 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf"] Nov 01 07:00:44.455594 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:00:44.455569 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3328298e_18f1_47b9_8577_3a3344f35155.slice/crio-a1919f5eb4bf40cac7fad3ae8ea23efdae259709e634638b7c135ab570b656b6 WatchSource:0}: Error finding container a1919f5eb4bf40cac7fad3ae8ea23efdae259709e634638b7c135ab570b656b6: Status 404 returned error can't find the container with id a1919f5eb4bf40cac7fad3ae8ea23efdae259709e634638b7c135ab570b656b6 Nov 01 07:00:45.219417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:45.219387 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" event={"ID":"3328298e-18f1-47b9-8577-3a3344f35155","Type":"ContainerStarted","Data":"a1919f5eb4bf40cac7fad3ae8ea23efdae259709e634638b7c135ab570b656b6"} Nov 01 07:00:46.258300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:46.255123 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" event={"ID":"3328298e-18f1-47b9-8577-3a3344f35155","Type":"ContainerStarted","Data":"8d3a90759dcd1ec71882a24f6d3e7bd2c8e0c1875cf91b2df0bd4d8ebca27e7c"} Nov 01 07:00:46.258300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:46.256185 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:00:46.376709 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:46.376645 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" podStartSLOduration=2.387799755 podStartE2EDuration="3.376625175s" totalImagesPullingTime="988.82542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:00:46.371350673 +0000 UTC m=+3061.924139869" watchObservedRunningTime="2026-11-01 07:00:46.376625175 +0000 UTC m=+3061.929414370" Nov 01 07:00:53.273347 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:00:53.273310 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:01:07.593195 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:01:07.593163 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:01:27.599945 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:27.599914 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc"] Nov 01 07:01:32.601173 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:32.601116 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" podUID="61b0c163-5063-4404-8d8d-8f73d223d0ca" containerName="payload-processing" containerID="cri-o://60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595" gracePeriod=25 Nov 01 07:01:32.869131 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:32.869071 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 07:01:33.040333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.040265 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume\") pod \"61b0c163-5063-4404-8d8d-8f73d223d0ca\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " Nov 01 07:01:33.040508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.040368 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn\" (UniqueName: \"kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn\") pod \"61b0c163-5063-4404-8d8d-8f73d223d0ca\" (UID: \"61b0c163-5063-4404-8d8d-8f73d223d0ca\") " Nov 01 07:01:33.040712 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.040687 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume" pod "61b0c163-5063-4404-8d8d-8f73d223d0ca" (UID: "61b0c163-5063-4404-8d8d-8f73d223d0ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:01:33.042603 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.042584 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn" pod "61b0c163-5063-4404-8d8d-8f73d223d0ca" (UID: "61b0c163-5063-4404-8d8d-8f73d223d0ca"). InnerVolumeSpecName "kube-api-access-lxrmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:01:33.141693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.141619 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b0c163-5063-4404-8d8d-8f73d223d0ca-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:01:33.141693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.141647 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxrmn\" (UniqueName: \"kubernetes.io/projected/61b0c163-5063-4404-8d8d-8f73d223d0ca-kube-api-access-lxrmn\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:01:33.561456 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.561362 4914 generic.go:356] "Generic (PLEG): container finished" podID="61b0c163-5063-4404-8d8d-8f73d223d0ca" containerID="60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595" exitCode=0 Nov 01 07:01:33.561456 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.561410 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" Nov 01 07:01:33.561456 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.561448 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" event={"ID":"61b0c163-5063-4404-8d8d-8f73d223d0ca","Type":"ContainerDied","Data":"60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595"} Nov 01 07:01:33.561714 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.561500 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc" event={"ID":"61b0c163-5063-4404-8d8d-8f73d223d0ca","Type":"ContainerDied","Data":"a343f0ea17125b31a17dbb002bf08a3f6b8879ce27f9dbfe746f4956c34efff0"} Nov 01 07:01:33.561714 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.561523 4914 scope.go:122] "RemoveContainer" containerID="60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595" Nov 01 07:01:33.572023 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.572005 4914 scope.go:122] "RemoveContainer" containerID="60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595" Nov 01 07:01:33.572297 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:01:33.572258 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595\": container with ID starting with 60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595 not found: ID does not exist" containerID="60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595" Nov 01 07:01:33.572356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.572307 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595"} err="failed to get container status \"60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595\": rpc error: code = NotFound desc = could not find container \"60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595\": container with ID starting with 60455cff5a953ace6a092901cf0696aeb3d8917fee6073c73bca8442de4fc595 not found: ID does not exist" Nov 01 07:01:33.725353 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.725326 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc"] Nov 01 07:01:33.774307 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:33.774256 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0cb4ef-cc64466f-gphkc"] Nov 01 07:01:34.617327 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:34.617294 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b0c163-5063-4404-8d8d-8f73d223d0ca" path="/var/lib/kubelet/pods/61b0c163-5063-4404-8d8d-8f73d223d0ca/volumes" Nov 01 07:01:37.594330 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:01:37.594270 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:01:42.723129 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:42.723097 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp"] Nov 01 07:01:47.724178 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:47.724143 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" podUID="3854de33-4b34-44d3-a618-4f6630aad360" containerName="payload-pre-processing" containerID="cri-o://715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041" gracePeriod=25 Nov 01 07:01:51.670228 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:01:51.670181 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" podUID="3854de33-4b34-44d3-a618-4f6630aad360" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:02:01.670323 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:01.670268 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" podUID="3854de33-4b34-44d3-a618-4f6630aad360" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:02:07.593121 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:02:07.593079 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:02:08.001209 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.001184 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 07:02:08.116849 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.116800 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume\") pod \"3854de33-4b34-44d3-a618-4f6630aad360\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " Nov 01 07:02:08.117030 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.116979 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq\" (UniqueName: \"kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq\") pod \"3854de33-4b34-44d3-a618-4f6630aad360\" (UID: \"3854de33-4b34-44d3-a618-4f6630aad360\") " Nov 01 07:02:08.117364 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.117339 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume" pod "3854de33-4b34-44d3-a618-4f6630aad360" (UID: "3854de33-4b34-44d3-a618-4f6630aad360"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:08.117364 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.117564 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3854de33-4b34-44d3-a618-4f6630aad360-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:08.119609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.119584 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq" pod "3854de33-4b34-44d3-a618-4f6630aad360" (UID: "3854de33-4b34-44d3-a618-4f6630aad360"). InnerVolumeSpecName "kube-api-access-7d2dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:08.218888 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.218860 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d2dq\" (UniqueName: \"kubernetes.io/projected/3854de33-4b34-44d3-a618-4f6630aad360-kube-api-access-7d2dq\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:08.802261 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.802225 4914 generic.go:356] "Generic (PLEG): container finished" podID="3854de33-4b34-44d3-a618-4f6630aad360" containerID="715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041" exitCode=0 Nov 01 07:02:08.802687 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.802266 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" Nov 01 07:02:08.802687 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.802310 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" event={"ID":"3854de33-4b34-44d3-a618-4f6630aad360","Type":"ContainerDied","Data":"715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041"} Nov 01 07:02:08.802687 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.802359 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp" event={"ID":"3854de33-4b34-44d3-a618-4f6630aad360","Type":"ContainerDied","Data":"fbbdab284c1f275e34fb05260f383e1e7d416c091bde51ff7f30e4f8e6dc842c"} Nov 01 07:02:08.802687 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.802379 4914 scope.go:122] "RemoveContainer" containerID="715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041" Nov 01 07:02:08.812174 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.812157 4914 scope.go:122] "RemoveContainer" containerID="715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041" Nov 01 07:02:08.812479 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:02:08.812460 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041\": container with ID starting with 715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041 not found: ID does not exist" containerID="715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041" Nov 01 07:02:08.812532 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.812488 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041"} err="failed to get container status \"715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041\": rpc error: code = NotFound desc = could not find container \"715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041\": container with ID starting with 715089b4a9971feef907c993eee5519ddb389792e7264c4c25be0edb0f5c8041 not found: ID does not exist" Nov 01 07:02:08.965941 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.965868 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp"] Nov 01 07:02:08.968776 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:08.968755 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c831a4c0-66fbbdcdf5-qjvzp"] Nov 01 07:02:10.617835 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:10.617801 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3854de33-4b34-44d3-a618-4f6630aad360" path="/var/lib/kubelet/pods/3854de33-4b34-44d3-a618-4f6630aad360/volumes" Nov 01 07:02:15.406048 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:15.406010 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq"] Nov 01 07:02:15.406559 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:15.406245 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" containerID="cri-o://a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333" gracePeriod=30 Nov 01 07:02:18.215777 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:18.215739 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-d9bdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 07:02:18.216224 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:18.215820 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:02:20.687163 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.687103 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 07:02:20.758143 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758117 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758169 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758194 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758216 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758373 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758399 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758410 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758452 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758482 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758600 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758621 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy\") pod \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\" (UID: \"48f58a99-b7a8-4c80-bfe1-e65c5f593514\") " Nov 01 07:02:20.758685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758553 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758893 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758759 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.758893 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.758841 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:20.759104 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.759083 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.759109 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.759126 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.759138 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.759372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.759151 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.760589 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.760565 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:20.760681 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.760643 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:02:20.760835 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.760799 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "kube-api-access-h59wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:20.760925 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.760881 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy" pod "48f58a99-b7a8-4c80-bfe1-e65c5f593514" (UID: "48f58a99-b7a8-4c80-bfe1-e65c5f593514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:02:20.860054 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.860025 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.860179 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.860053 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.860179 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.860075 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48f58a99-b7a8-4c80-bfe1-e65c5f593514-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.860179 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.860084 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59wm\" (UniqueName: \"kubernetes.io/projected/48f58a99-b7a8-4c80-bfe1-e65c5f593514-kube-api-access-h59wm\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:20.881708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.881677 4914 generic.go:356] "Generic (PLEG): container finished" podID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerID="a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333" exitCode=0 Nov 01 07:02:20.881858 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.881733 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" Nov 01 07:02:20.881858 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.881763 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" event={"ID":"48f58a99-b7a8-4c80-bfe1-e65c5f593514","Type":"ContainerDied","Data":"a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333"} Nov 01 07:02:20.881858 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.881819 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq" event={"ID":"48f58a99-b7a8-4c80-bfe1-e65c5f593514","Type":"ContainerDied","Data":"bce6f5badc4d2e50025823b4e011c0557be0ef3e96cb5bad7c2d763e6a2fd889"} Nov 01 07:02:20.881858 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.881844 4914 scope.go:122] "RemoveContainer" containerID="a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333" Nov 01 07:02:20.893660 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.893642 4914 scope.go:122] "RemoveContainer" containerID="a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333" Nov 01 07:02:20.893937 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:02:20.893919 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333\": container with ID starting with a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333 not found: ID does not exist" containerID="a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333" Nov 01 07:02:20.894013 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:20.893948 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333"} err="failed to get container status \"a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333\": rpc error: code = NotFound desc = could not find container \"a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333\": container with ID starting with a67777ec309411b58dae262f1a68a30009dc9a3062fc3f40052ea774cbcb2333 not found: ID does not exist" Nov 01 07:02:21.045767 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:21.045679 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq"] Nov 01 07:02:21.049223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:21.049196 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-d9bdq"] Nov 01 07:02:22.618160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:22.618120 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" path="/var/lib/kubelet/pods/48f58a99-b7a8-4c80-bfe1-e65c5f593514/volumes" Nov 01 07:02:28.373746 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:28.373524 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x"] Nov 01 07:02:33.374005 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.373952 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" podUID="0f50f78f-ecdb-4a95-8746-7aa30b9c4044" containerName="payload-pre-processing" containerID="cri-o://e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484" gracePeriod=25 Nov 01 07:02:33.655353 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.655325 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 07:02:33.711335 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.711307 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume\") pod \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " Nov 01 07:02:33.711513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.711359 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9\" (UniqueName: \"kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9\") pod \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\" (UID: \"0f50f78f-ecdb-4a95-8746-7aa30b9c4044\") " Nov 01 07:02:33.711705 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.711670 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume" pod "0f50f78f-ecdb-4a95-8746-7aa30b9c4044" (UID: "0f50f78f-ecdb-4a95-8746-7aa30b9c4044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:33.712130 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.712091 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:33.714064 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.714043 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9" pod "0f50f78f-ecdb-4a95-8746-7aa30b9c4044" (UID: "0f50f78f-ecdb-4a95-8746-7aa30b9c4044"). InnerVolumeSpecName "kube-api-access-lqgz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:33.812974 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.812937 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqgz9\" (UniqueName: \"kubernetes.io/projected/0f50f78f-ecdb-4a95-8746-7aa30b9c4044-kube-api-access-lqgz9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:33.966632 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.966545 4914 generic.go:356] "Generic (PLEG): container finished" podID="0f50f78f-ecdb-4a95-8746-7aa30b9c4044" containerID="e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484" exitCode=0 Nov 01 07:02:33.966632 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.966591 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" Nov 01 07:02:33.966632 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.966604 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" event={"ID":"0f50f78f-ecdb-4a95-8746-7aa30b9c4044","Type":"ContainerDied","Data":"e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484"} Nov 01 07:02:33.966926 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.966658 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x" event={"ID":"0f50f78f-ecdb-4a95-8746-7aa30b9c4044","Type":"ContainerDied","Data":"1060af341b13e4e4667fe54b32ddd929fba206144ad8dd1fff83b479cfad5802"} Nov 01 07:02:33.966926 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.966684 4914 scope.go:122] "RemoveContainer" containerID="e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484" Nov 01 07:02:33.977627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.977609 4914 scope.go:122] "RemoveContainer" containerID="e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484" Nov 01 07:02:33.977878 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:02:33.977862 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484\": container with ID starting with e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484 not found: ID does not exist" containerID="e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484" Nov 01 07:02:33.977940 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:33.977885 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484"} err="failed to get container status \"e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484\": rpc error: code = NotFound desc = could not find container \"e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484\": container with ID starting with e56a41ea73ec55584b6f4b1bfde306dfd54887fc611a7eebe5fea6f717fa9484 not found: ID does not exist" Nov 01 07:02:34.129763 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:34.129738 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x"] Nov 01 07:02:34.137432 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:34.137404 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-3db672-c5cfb8f7f-k8b4x"] Nov 01 07:02:34.618970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:34.618940 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f50f78f-ecdb-4a95-8746-7aa30b9c4044" path="/var/lib/kubelet/pods/0f50f78f-ecdb-4a95-8746-7aa30b9c4044/volumes" Nov 01 07:02:37.594249 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:02:37.594215 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:02:37.700868 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:37.700842 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f"] Nov 01 07:02:37.701036 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:37.701012 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" podUID="c9873d7a-fc07-4dfa-8978-f560226a63c0" containerName="maas-api" containerID="cri-o://3a471feffb66dd069c3c8386360c025d418980d30dfab314c471a1bda14b1aed" gracePeriod=30 Nov 01 07:02:37.992651 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:37.992624 4914 generic.go:356] "Generic (PLEG): container finished" podID="c9873d7a-fc07-4dfa-8978-f560226a63c0" containerID="3a471feffb66dd069c3c8386360c025d418980d30dfab314c471a1bda14b1aed" exitCode=0 Nov 01 07:02:37.992772 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:37.992668 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" event={"ID":"c9873d7a-fc07-4dfa-8978-f560226a63c0","Type":"ContainerDied","Data":"3a471feffb66dd069c3c8386360c025d418980d30dfab314c471a1bda14b1aed"} Nov 01 07:02:37.992772 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:37.992710 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" event={"ID":"c9873d7a-fc07-4dfa-8978-f560226a63c0","Type":"ContainerDied","Data":"ed914b784e3fdeefbe5e9912213bee0e865b7061e440478008d70913db9fbd24"} Nov 01 07:02:37.992772 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:37.992726 4914 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed914b784e3fdeefbe5e9912213bee0e865b7061e440478008d70913db9fbd24" Nov 01 07:02:38.003355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.003335 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 07:02:38.055879 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.055851 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca\") pod \"c9873d7a-fc07-4dfa-8978-f560226a63c0\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " Nov 01 07:02:38.055979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.055918 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h\" (UniqueName: \"kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h\") pod \"c9873d7a-fc07-4dfa-8978-f560226a63c0\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " Nov 01 07:02:38.056034 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.056017 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs\") pod \"c9873d7a-fc07-4dfa-8978-f560226a63c0\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " Nov 01 07:02:38.056102 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.056087 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\") pod \"c9873d7a-fc07-4dfa-8978-f560226a63c0\" (UID: \"c9873d7a-fc07-4dfa-8978-f560226a63c0\") " Nov 01 07:02:38.056196 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.056171 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca" pod "c9873d7a-fc07-4dfa-8978-f560226a63c0" (UID: "c9873d7a-fc07-4dfa-8978-f560226a63c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:02:38.056492 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.056472 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9873d7a-fc07-4dfa-8978-f560226a63c0-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:38.058239 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.058215 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls" pod "c9873d7a-fc07-4dfa-8978-f560226a63c0" (UID: "c9873d7a-fc07-4dfa-8978-f560226a63c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:02:38.058357 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.058252 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h" pod "c9873d7a-fc07-4dfa-8978-f560226a63c0" (UID: "c9873d7a-fc07-4dfa-8978-f560226a63c0"). InnerVolumeSpecName "kube-api-access-79f2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:02:38.058357 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.058261 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs" pod "c9873d7a-fc07-4dfa-8978-f560226a63c0" (UID: "c9873d7a-fc07-4dfa-8978-f560226a63c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:02:38.157840 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.157803 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:38.157840 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.157836 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79f2h\" (UniqueName: \"kubernetes.io/projected/c9873d7a-fc07-4dfa-8978-f560226a63c0-kube-api-access-79f2h\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:38.157840 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.157845 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9873d7a-fc07-4dfa-8978-f560226a63c0-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:02:38.999334 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:38.999243 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f" Nov 01 07:02:39.162096 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:39.162064 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f"] Nov 01 07:02:39.212309 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:39.212267 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0cb4ef-595bc7cd6-5tt6f"] Nov 01 07:02:40.618267 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:02:40.618231 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9873d7a-fc07-4dfa-8978-f560226a63c0" path="/var/lib/kubelet/pods/c9873d7a-fc07-4dfa-8978-f560226a63c0/volumes" Nov 01 07:03:07.593292 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:03:07.593191 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:03:21.199318 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.199262 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp"] Nov 01 07:03:21.199715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.199514 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" podUID="9d434462-1888-45d1-a387-e0eea3df84b6" containerName="maas-api" containerID="cri-o://df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3" gracePeriod=30 Nov 01 07:03:21.494130 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.494102 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 07:03:21.645925 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.645873 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs\") pod \"9d434462-1888-45d1-a387-e0eea3df84b6\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " Nov 01 07:03:21.646139 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.646048 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\") pod \"9d434462-1888-45d1-a387-e0eea3df84b6\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " Nov 01 07:03:21.646215 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.646176 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca\") pod \"9d434462-1888-45d1-a387-e0eea3df84b6\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " Nov 01 07:03:21.646259 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.646236 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf\" (UniqueName: \"kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf\") pod \"9d434462-1888-45d1-a387-e0eea3df84b6\" (UID: \"9d434462-1888-45d1-a387-e0eea3df84b6\") " Nov 01 07:03:21.646629 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.646598 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca" pod "9d434462-1888-45d1-a387-e0eea3df84b6" (UID: "9d434462-1888-45d1-a387-e0eea3df84b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:03:21.646785 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.646766 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d434462-1888-45d1-a387-e0eea3df84b6-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:21.648692 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.648654 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls" pod "9d434462-1888-45d1-a387-e0eea3df84b6" (UID: "9d434462-1888-45d1-a387-e0eea3df84b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:21.649032 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.649002 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs" pod "9d434462-1888-45d1-a387-e0eea3df84b6" (UID: "9d434462-1888-45d1-a387-e0eea3df84b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:21.649209 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.649180 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf" pod "9d434462-1888-45d1-a387-e0eea3df84b6" (UID: "9d434462-1888-45d1-a387-e0eea3df84b6"). InnerVolumeSpecName "kube-api-access-7wxbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:21.751524 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.750740 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:21.751524 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.750780 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d434462-1888-45d1-a387-e0eea3df84b6-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:21.751524 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:21.750795 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxbf\" (UniqueName: \"kubernetes.io/projected/9d434462-1888-45d1-a387-e0eea3df84b6-kube-api-access-7wxbf\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:22.266977 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.266945 4914 generic.go:356] "Generic (PLEG): container finished" podID="9d434462-1888-45d1-a387-e0eea3df84b6" containerID="df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3" exitCode=0 Nov 01 07:03:22.267436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.266997 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" Nov 01 07:03:22.267436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.267035 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" event={"ID":"9d434462-1888-45d1-a387-e0eea3df84b6","Type":"ContainerDied","Data":"df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3"} Nov 01 07:03:22.267436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.267106 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp" event={"ID":"9d434462-1888-45d1-a387-e0eea3df84b6","Type":"ContainerDied","Data":"66c16685c621fb566da862f4f5e7e916affaa13e77f21f278a95a29716f38179"} Nov 01 07:03:22.267436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.267149 4914 scope.go:122] "RemoveContainer" containerID="df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3" Nov 01 07:03:22.277849 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.277835 4914 scope.go:122] "RemoveContainer" containerID="df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3" Nov 01 07:03:22.278135 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:03:22.278118 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3\": container with ID starting with df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3 not found: ID does not exist" containerID="df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3" Nov 01 07:03:22.278198 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.278145 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3"} err="failed to get container status \"df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3\": rpc error: code = NotFound desc = could not find container \"df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3\": container with ID starting with df7839733bfadd3833ade30ab8d61fa0b3c2ceef7762a38ba0686b932b8028e3 not found: ID does not exist" Nov 01 07:03:22.430437 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.430404 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp"] Nov 01 07:03:22.433243 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.433216 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3db672-9d49b9589-jnpkp"] Nov 01 07:03:22.618495 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:22.618467 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d434462-1888-45d1-a387-e0eea3df84b6" path="/var/lib/kubelet/pods/9d434462-1888-45d1-a387-e0eea3df84b6/volumes" Nov 01 07:03:30.947411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:30.947368 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76"] Nov 01 07:03:32.125895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.125862 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf"] Nov 01 07:03:32.126325 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.126101 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" podUID="3328298e-18f1-47b9-8577-3a3344f35155" containerName="maas-api" containerID="cri-o://8d3a90759dcd1ec71882a24f6d3e7bd2c8e0c1875cf91b2df0bd4d8ebca27e7c" gracePeriod=30 Nov 01 07:03:32.338025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.337995 4914 generic.go:356] "Generic (PLEG): container finished" podID="3328298e-18f1-47b9-8577-3a3344f35155" containerID="8d3a90759dcd1ec71882a24f6d3e7bd2c8e0c1875cf91b2df0bd4d8ebca27e7c" exitCode=0 Nov 01 07:03:32.338192 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.338073 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" event={"ID":"3328298e-18f1-47b9-8577-3a3344f35155","Type":"ContainerDied","Data":"8d3a90759dcd1ec71882a24f6d3e7bd2c8e0c1875cf91b2df0bd4d8ebca27e7c"} Nov 01 07:03:32.437544 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.437518 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:03:32.572397 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.572359 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca\") pod \"3328298e-18f1-47b9-8577-3a3344f35155\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " Nov 01 07:03:32.572583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.572482 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr\" (UniqueName: \"kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr\") pod \"3328298e-18f1-47b9-8577-3a3344f35155\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " Nov 01 07:03:32.572583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.572540 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs\") pod \"3328298e-18f1-47b9-8577-3a3344f35155\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " Nov 01 07:03:32.572715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.572598 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\") pod \"3328298e-18f1-47b9-8577-3a3344f35155\" (UID: \"3328298e-18f1-47b9-8577-3a3344f35155\") " Nov 01 07:03:32.572789 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.572752 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca" pod "3328298e-18f1-47b9-8577-3a3344f35155" (UID: "3328298e-18f1-47b9-8577-3a3344f35155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:03:32.573079 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.573055 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3328298e-18f1-47b9-8577-3a3344f35155-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:32.574856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.574828 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls" pod "3328298e-18f1-47b9-8577-3a3344f35155" (UID: "3328298e-18f1-47b9-8577-3a3344f35155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:32.575260 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.575235 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs" pod "3328298e-18f1-47b9-8577-3a3344f35155" (UID: "3328298e-18f1-47b9-8577-3a3344f35155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:03:32.575394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.575324 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr" pod "3328298e-18f1-47b9-8577-3a3344f35155" (UID: "3328298e-18f1-47b9-8577-3a3344f35155"). InnerVolumeSpecName "kube-api-access-bfzlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:32.674253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.674165 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzlr\" (UniqueName: \"kubernetes.io/projected/3328298e-18f1-47b9-8577-3a3344f35155-kube-api-access-bfzlr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:32.674253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.674200 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:32.674253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:32.674216 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3328298e-18f1-47b9-8577-3a3344f35155-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:33.345164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:33.345138 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" Nov 01 07:03:33.345617 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:33.345136 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf" event={"ID":"3328298e-18f1-47b9-8577-3a3344f35155","Type":"ContainerDied","Data":"a1919f5eb4bf40cac7fad3ae8ea23efdae259709e634638b7c135ab570b656b6"} Nov 01 07:03:33.345617 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:33.345296 4914 scope.go:122] "RemoveContainer" containerID="8d3a90759dcd1ec71882a24f6d3e7bd2c8e0c1875cf91b2df0bd4d8ebca27e7c" Nov 01 07:03:33.510351 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:33.510315 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf"] Nov 01 07:03:33.559820 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:33.559780 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c52bfa-5d4567ff55-8gwnf"] Nov 01 07:03:34.616713 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:34.616678 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3328298e-18f1-47b9-8577-3a3344f35155" path="/var/lib/kubelet/pods/3328298e-18f1-47b9-8577-3a3344f35155/volumes" Nov 01 07:03:35.948620 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:35.948576 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" podUID="89d5cda5-cac4-4296-bb88-d74ace000331" containerName="payload-processing" containerID="cri-o://f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0" gracePeriod=25 Nov 01 07:03:36.218409 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.218337 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 07:03:36.313238 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.313202 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg\" (UniqueName: \"kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg\") pod \"89d5cda5-cac4-4296-bb88-d74ace000331\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " Nov 01 07:03:36.313464 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.313384 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume\") pod \"89d5cda5-cac4-4296-bb88-d74ace000331\" (UID: \"89d5cda5-cac4-4296-bb88-d74ace000331\") " Nov 01 07:03:36.313744 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.313723 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume" pod "89d5cda5-cac4-4296-bb88-d74ace000331" (UID: "89d5cda5-cac4-4296-bb88-d74ace000331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:03:36.315506 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.315483 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg" pod "89d5cda5-cac4-4296-bb88-d74ace000331" (UID: "89d5cda5-cac4-4296-bb88-d74ace000331"). InnerVolumeSpecName "kube-api-access-59sjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:03:36.366931 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.366884 4914 generic.go:356] "Generic (PLEG): container finished" podID="89d5cda5-cac4-4296-bb88-d74ace000331" containerID="f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0" exitCode=0 Nov 01 07:03:36.367117 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.366993 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" Nov 01 07:03:36.367117 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.367028 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" event={"ID":"89d5cda5-cac4-4296-bb88-d74ace000331","Type":"ContainerDied","Data":"f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0"} Nov 01 07:03:36.367117 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.367091 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76" event={"ID":"89d5cda5-cac4-4296-bb88-d74ace000331","Type":"ContainerDied","Data":"97a6c806d7a3f771585fc8d90a56181c705c82cc25902e6ab84b5cf4dea617c1"} Nov 01 07:03:36.367308 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.367120 4914 scope.go:122] "RemoveContainer" containerID="f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0" Nov 01 07:03:36.377953 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.377929 4914 scope.go:122] "RemoveContainer" containerID="f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0" Nov 01 07:03:36.378200 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:03:36.378182 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0\": container with ID starting with f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0 not found: ID does not exist" containerID="f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0" Nov 01 07:03:36.378269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.378207 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0"} err="failed to get container status \"f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0\": rpc error: code = NotFound desc = could not find container \"f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0\": container with ID starting with f665e8c619c3564b8306c79098f4f49fe3fedfe31d8279399324ba478d3341f0 not found: ID does not exist" Nov 01 07:03:36.415190 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.415146 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5cda5-cac4-4296-bb88-d74ace000331-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:36.415190 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.415182 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59sjg\" (UniqueName: \"kubernetes.io/projected/89d5cda5-cac4-4296-bb88-d74ace000331-kube-api-access-59sjg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:03:36.532232 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.532100 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76"] Nov 01 07:03:36.543656 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.543621 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c52bfa-78d4465dfc-rgz76"] Nov 01 07:03:36.617441 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:03:36.617408 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d5cda5-cac4-4296-bb88-d74ace000331" path="/var/lib/kubelet/pods/89d5cda5-cac4-4296-bb88-d74ace000331/volumes" Nov 01 07:03:37.593403 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:03:37.593364 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:04:03.960779 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:03.960741 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb"] Nov 01 07:04:07.593829 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:07.593783 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:04:08.961405 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:08.961370 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerName="payload-pre-processing" containerID="cri-o://cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9" gracePeriod=25 Nov 01 07:04:12.143162 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:12.143125 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:04:13.824834 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:13.824805 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg"] Nov 01 07:04:13.825310 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:13.825057 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" podUID="393cafe2-a612-4725-b031-fd45c6496b80" containerName="maas-api" containerID="cri-o://cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69" gracePeriod=30 Nov 01 07:04:14.118707 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.118679 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 07:04:14.152747 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.152719 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs\") pod \"393cafe2-a612-4725-b031-fd45c6496b80\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " Nov 01 07:04:14.152874 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.152767 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca\") pod \"393cafe2-a612-4725-b031-fd45c6496b80\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " Nov 01 07:04:14.152874 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.152852 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb\" (UniqueName: \"kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb\") pod \"393cafe2-a612-4725-b031-fd45c6496b80\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " Nov 01 07:04:14.152990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.152923 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\") pod \"393cafe2-a612-4725-b031-fd45c6496b80\" (UID: \"393cafe2-a612-4725-b031-fd45c6496b80\") " Nov 01 07:04:14.153197 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.153173 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca" pod "393cafe2-a612-4725-b031-fd45c6496b80" (UID: "393cafe2-a612-4725-b031-fd45c6496b80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:14.155018 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.154987 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs" pod "393cafe2-a612-4725-b031-fd45c6496b80" (UID: "393cafe2-a612-4725-b031-fd45c6496b80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:04:14.155105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.155027 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls" pod "393cafe2-a612-4725-b031-fd45c6496b80" (UID: "393cafe2-a612-4725-b031-fd45c6496b80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:04:14.155105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.155041 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb" pod "393cafe2-a612-4725-b031-fd45c6496b80" (UID: "393cafe2-a612-4725-b031-fd45c6496b80"). InnerVolumeSpecName "kube-api-access-8lggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:14.254752 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.254717 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:14.254752 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.254755 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393cafe2-a612-4725-b031-fd45c6496b80-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:14.255003 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.254771 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lggb\" (UniqueName: \"kubernetes.io/projected/393cafe2-a612-4725-b031-fd45c6496b80-kube-api-access-8lggb\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:14.255003 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.254785 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393cafe2-a612-4725-b031-fd45c6496b80-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:14.637803 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.637769 4914 generic.go:356] "Generic (PLEG): container finished" podID="393cafe2-a612-4725-b031-fd45c6496b80" containerID="cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69" exitCode=0 Nov 01 07:04:14.637940 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.637802 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" event={"ID":"393cafe2-a612-4725-b031-fd45c6496b80","Type":"ContainerDied","Data":"cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69"} Nov 01 07:04:14.637940 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.637830 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" Nov 01 07:04:14.637940 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.637854 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg" event={"ID":"393cafe2-a612-4725-b031-fd45c6496b80","Type":"ContainerDied","Data":"8d24535fd0ee16acf9ad1accd27327ca8c54529115e588d941f032ccb8ba4829"} Nov 01 07:04:14.637940 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.637878 4914 scope.go:122] "RemoveContainer" containerID="cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69" Nov 01 07:04:14.647690 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.647669 4914 scope.go:122] "RemoveContainer" containerID="cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69" Nov 01 07:04:14.647954 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:14.647932 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69\": container with ID starting with cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69 not found: ID does not exist" containerID="cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69" Nov 01 07:04:14.648054 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.647958 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69"} err="failed to get container status \"cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69\": rpc error: code = NotFound desc = could not find container \"cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69\": container with ID starting with cef4140a9a4f865a76ec8b182df8d8e79a4a5e6742ebe305dfc1de2c7c90dc69 not found: ID does not exist" Nov 01 07:04:14.820708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.820674 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg"] Nov 01 07:04:14.869873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:14.869834 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c79abe-686cfcc946-4mxsg"] Nov 01 07:04:16.617048 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:16.617009 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393cafe2-a612-4725-b031-fd45c6496b80" path="/var/lib/kubelet/pods/393cafe2-a612-4725-b031-fd45c6496b80/volumes" Nov 01 07:04:22.143986 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:22.143941 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:04:29.220981 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.220920 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 07:04:29.362889 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.362859 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume\") pod \"0a7e977e-6407-40b1-9d9a-656750d3e96a\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " Nov 01 07:04:29.363081 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.362896 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk\" (UniqueName: \"kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk\") pod \"0a7e977e-6407-40b1-9d9a-656750d3e96a\" (UID: \"0a7e977e-6407-40b1-9d9a-656750d3e96a\") " Nov 01 07:04:29.363354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.363322 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume" pod "0a7e977e-6407-40b1-9d9a-656750d3e96a" (UID: "0a7e977e-6407-40b1-9d9a-656750d3e96a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:29.365302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.365259 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk" pod "0a7e977e-6407-40b1-9d9a-656750d3e96a" (UID: "0a7e977e-6407-40b1-9d9a-656750d3e96a"). InnerVolumeSpecName "kube-api-access-d2vbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:29.464794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.464760 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a7e977e-6407-40b1-9d9a-656750d3e96a-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:29.464794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.464791 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vbk\" (UniqueName: \"kubernetes.io/projected/0a7e977e-6407-40b1-9d9a-656750d3e96a-kube-api-access-d2vbk\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:29.732798 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.732759 4914 generic.go:356] "Generic (PLEG): container finished" podID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerID="cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9" exitCode=0 Nov 01 07:04:29.732976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.732810 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" Nov 01 07:04:29.732976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.732821 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" event={"ID":"0a7e977e-6407-40b1-9d9a-656750d3e96a","Type":"ContainerDied","Data":"cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9"} Nov 01 07:04:29.732976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.732881 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb" event={"ID":"0a7e977e-6407-40b1-9d9a-656750d3e96a","Type":"ContainerDied","Data":"cd1bbd3071cb7eae9d42d413f6bd86653bff2f7dbd75cbbd13627bddd784eb62"} Nov 01 07:04:29.732976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.732911 4914 scope.go:122] "RemoveContainer" containerID="cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9" Nov 01 07:04:29.743048 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.743029 4914 scope.go:122] "RemoveContainer" containerID="cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9" Nov 01 07:04:29.743370 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:29.743345 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9\": container with ID starting with cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9 not found: ID does not exist" containerID="cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9" Nov 01 07:04:29.743471 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.743377 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9"} err="failed to get container status \"cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9\": rpc error: code = NotFound desc = could not find container \"cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9\": container with ID starting with cd926bb961e5ba0356d5fc9c492ce308c8eee8af732efdcd62f527338b79dbd9 not found: ID does not exist" Nov 01 07:04:29.908992 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.908956 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb"] Nov 01 07:04:29.921580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:29.921555 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c79abe-74c56b847f-nknsb"] Nov 01 07:04:30.620721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:30.620686 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" path="/var/lib/kubelet/pods/0a7e977e-6407-40b1-9d9a-656750d3e96a/volumes" Nov 01 07:04:31.819421 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:31.819386 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf"] Nov 01 07:04:31.923604 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:31.923574 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r"] Nov 01 07:04:36.820122 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:36.820073 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" podUID="22c047da-a881-4066-ad57-73e37e66980c" containerName="payload-processing" containerID="cri-o://acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07" gracePeriod=25 Nov 01 07:04:36.923878 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:36.923834 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" podUID="284136a9-8bc0-4a17-8c61-a0bbc1b87f90" containerName="payload-pre-processing" containerID="cri-o://916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456" gracePeriod=25 Nov 01 07:04:37.200625 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.200596 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 07:04:37.274648 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.274615 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume\") pod \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " Nov 01 07:04:37.274804 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.274787 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9\" (UniqueName: \"kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9\") pod \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\" (UID: \"284136a9-8bc0-4a17-8c61-a0bbc1b87f90\") " Nov 01 07:04:37.275027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.275000 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume" pod "284136a9-8bc0-4a17-8c61-a0bbc1b87f90" (UID: "284136a9-8bc0-4a17-8c61-a0bbc1b87f90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:37.275142 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.275124 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:37.276937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.276915 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9" pod "284136a9-8bc0-4a17-8c61-a0bbc1b87f90" (UID: "284136a9-8bc0-4a17-8c61-a0bbc1b87f90"). InnerVolumeSpecName "kube-api-access-lztv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:37.376371 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.376303 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lztv9\" (UniqueName: \"kubernetes.io/projected/284136a9-8bc0-4a17-8c61-a0bbc1b87f90-kube-api-access-lztv9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:37.593002 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:37.592971 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:04:37.783127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.783054 4914 generic.go:356] "Generic (PLEG): container finished" podID="284136a9-8bc0-4a17-8c61-a0bbc1b87f90" containerID="916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456" exitCode=0 Nov 01 07:04:37.783127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.783102 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" Nov 01 07:04:37.783360 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.783132 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" event={"ID":"284136a9-8bc0-4a17-8c61-a0bbc1b87f90","Type":"ContainerDied","Data":"916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456"} Nov 01 07:04:37.783360 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.783182 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r" event={"ID":"284136a9-8bc0-4a17-8c61-a0bbc1b87f90","Type":"ContainerDied","Data":"020eeb118e8da2867ec57a36abb01b8933def83991edc570888198dc9367ac74"} Nov 01 07:04:37.783360 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.783205 4914 scope.go:122] "RemoveContainer" containerID="916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456" Nov 01 07:04:37.794049 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.794031 4914 scope.go:122] "RemoveContainer" containerID="916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456" Nov 01 07:04:37.794312 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:37.794293 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456\": container with ID starting with 916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456 not found: ID does not exist" containerID="916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456" Nov 01 07:04:37.794356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.794319 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456"} err="failed to get container status \"916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456\": rpc error: code = NotFound desc = could not find container \"916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456\": container with ID starting with 916e707fce0a747df2b7e1446292952c67db2b1e302783c688c1d8aa4c2ad456 not found: ID does not exist" Nov 01 07:04:37.989736 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:37.989703 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r"] Nov 01 07:04:38.001039 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:38.001014 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc6e48-6ff9c58855-5q79r"] Nov 01 07:04:38.617156 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:38.617123 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284136a9-8bc0-4a17-8c61-a0bbc1b87f90" path="/var/lib/kubelet/pods/284136a9-8bc0-4a17-8c61-a0bbc1b87f90/volumes" Nov 01 07:04:39.064508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.064426 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk"] Nov 01 07:04:39.064927 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.064642 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" podUID="2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" containerName="maas-api" containerID="cri-o://0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2" gracePeriod=30 Nov 01 07:04:39.347365 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.347331 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 07:04:39.398903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.398873 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs\") pod \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " Nov 01 07:04:39.399042 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.398947 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz\" (UniqueName: \"kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz\") pod \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " Nov 01 07:04:39.399042 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.399002 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\") pod \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " Nov 01 07:04:39.399166 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.399108 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca\") pod \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\" (UID: \"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a\") " Nov 01 07:04:39.399638 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.399542 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca" pod "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" (UID: "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:39.401104 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.401076 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs" pod "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" (UID: "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:04:39.401222 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.401185 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz" pod "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" (UID: "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a"). InnerVolumeSpecName "kube-api-access-xfnbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:39.401270 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.401214 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls" pod "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" (UID: "2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:04:39.500751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.500713 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnbz\" (UniqueName: \"kubernetes.io/projected/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-kube-api-access-xfnbz\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:39.500751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.500752 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:39.500956 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.500768 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:39.500956 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.500782 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:39.597356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.596959 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" podUID="22c047da-a881-4066-ad57-73e37e66980c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:04:39.797752 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.797722 4914 generic.go:356] "Generic (PLEG): container finished" podID="2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" containerID="0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2" exitCode=0 Nov 01 07:04:39.797928 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.797765 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" Nov 01 07:04:39.797928 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.797783 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" event={"ID":"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a","Type":"ContainerDied","Data":"0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2"} Nov 01 07:04:39.797928 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.797822 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk" event={"ID":"2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a","Type":"ContainerDied","Data":"64898fdf3f9e2f9e8a3fbc46642d136128be9e8f8bdadf6cfce9167190603151"} Nov 01 07:04:39.797928 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.797845 4914 scope.go:122] "RemoveContainer" containerID="0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2" Nov 01 07:04:39.808507 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.808490 4914 scope.go:122] "RemoveContainer" containerID="0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2" Nov 01 07:04:39.808770 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:39.808747 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2\": container with ID starting with 0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2 not found: ID does not exist" containerID="0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2" Nov 01 07:04:39.808818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.808780 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2"} err="failed to get container status \"0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2\": rpc error: code = NotFound desc = could not find container \"0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2\": container with ID starting with 0ad7a79969580ff7f01c9fdc04d0335bd151fb4aaa2856303c2d7c413fec33c2 not found: ID does not exist" Nov 01 07:04:39.961833 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.961752 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk"] Nov 01 07:04:39.965017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:39.964987 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-dc6e48-665bfd5f4d-r58pk"] Nov 01 07:04:40.616814 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:40.616784 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" path="/var/lib/kubelet/pods/2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a/volumes" Nov 01 07:04:45.021936 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.021908 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:04:45.034717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.034623 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:04:45.413265 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.413234 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn"] Nov 01 07:04:45.413476 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.413452 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" containerID="cri-o://c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4" gracePeriod=30 Nov 01 07:04:45.615338 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.615307 4914 scope.go:122] "RemoveContainer" containerID="3a471feffb66dd069c3c8386360c025d418980d30dfab314c471a1bda14b1aed" Nov 01 07:04:45.710921 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.710866 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-r9phn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 07:04:45.711024 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:45.710930 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:04:49.584957 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:49.584917 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" podUID="22c047da-a881-4066-ad57-73e37e66980c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:04:50.468800 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.468768 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh"] Nov 01 07:04:50.469339 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469324 4914 cpu_manager.go:414] "containerMap: removing container" podUID="284136a9-8bc0-4a17-8c61-a0bbc1b87f90" containerName="payload-pre-processing" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469340 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284136a9-8bc0-4a17-8c61-a0bbc1b87f90" containerName="payload-pre-processing" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469352 4914 cpu_manager.go:414] "containerMap: removing container" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469357 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469369 4914 cpu_manager.go:414] "containerMap: removing container" podUID="61b0c163-5063-4404-8d8d-8f73d223d0ca" containerName="payload-processing" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469375 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b0c163-5063-4404-8d8d-8f73d223d0ca" containerName="payload-processing" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469387 4914 cpu_manager.go:414] "containerMap: removing container" podUID="89d5cda5-cac4-4296-bb88-d74ace000331" containerName="payload-processing" Nov 01 07:04:50.469392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469392 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d5cda5-cac4-4296-bb88-d74ace000331" containerName="payload-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469399 4914 cpu_manager.go:414] "containerMap: removing container" podUID="9d434462-1888-45d1-a387-e0eea3df84b6" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469404 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d434462-1888-45d1-a387-e0eea3df84b6" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469411 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469415 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469424 4914 cpu_manager.go:414] "containerMap: removing container" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerName="payload-pre-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469428 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerName="payload-pre-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469449 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c9873d7a-fc07-4dfa-8978-f560226a63c0" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469454 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9873d7a-fc07-4dfa-8978-f560226a63c0" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469462 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3328298e-18f1-47b9-8577-3a3344f35155" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469466 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3328298e-18f1-47b9-8577-3a3344f35155" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469475 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3854de33-4b34-44d3-a618-4f6630aad360" containerName="payload-pre-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469479 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3854de33-4b34-44d3-a618-4f6630aad360" containerName="payload-pre-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469488 4914 cpu_manager.go:414] "containerMap: removing container" podUID="393cafe2-a612-4725-b031-fd45c6496b80" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469492 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393cafe2-a612-4725-b031-fd45c6496b80" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469500 4914 cpu_manager.go:414] "containerMap: removing container" podUID="0f50f78f-ecdb-4a95-8746-7aa30b9c4044" containerName="payload-pre-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469504 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f50f78f-ecdb-4a95-8746-7aa30b9c4044" containerName="payload-pre-processing" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469586 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9873d7a-fc07-4dfa-8978-f560226a63c0" containerName="maas-api" Nov 01 07:04:50.469591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469594 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3328298e-18f1-47b9-8577-3a3344f35155" containerName="maas-api" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469600 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3854de33-4b34-44d3-a618-4f6630aad360" containerName="payload-pre-processing" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469607 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="393cafe2-a612-4725-b031-fd45c6496b80" containerName="maas-api" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469614 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f50f78f-ecdb-4a95-8746-7aa30b9c4044" containerName="payload-pre-processing" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469620 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="284136a9-8bc0-4a17-8c61-a0bbc1b87f90" containerName="payload-pre-processing" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469624 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f58a99-b7a8-4c80-bfe1-e65c5f593514" containerName="istio-proxy" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469629 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b0c163-5063-4404-8d8d-8f73d223d0ca" containerName="payload-processing" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469636 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d5cda5-cac4-4296-bb88-d74ace000331" containerName="payload-processing" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469641 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d434462-1888-45d1-a387-e0eea3df84b6" containerName="maas-api" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469647 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd95b96-99e7-45f6-b4e1-38f6ef9eb63a" containerName="maas-api" Nov 01 07:04:50.470095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.469652 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7e977e-6407-40b1-9d9a-656750d3e96a" containerName="payload-pre-processing" Nov 01 07:04:50.472881 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.472865 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.512579 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.512552 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.512671 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.512594 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.512770 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.512748 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkr7h\" (UniqueName: \"kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.512818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.512788 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.565380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.565347 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c831a4c0-serving-cert\"" Nov 01 07:04:50.568627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.568600 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh"] Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.613937 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.614041 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkr7h\" (UniqueName: \"kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.614070 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.614228 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:50.614370 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-serving-cert: secret "maas-api-e2e-mt-c831a4c0-serving-cert" not found Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:50.614442 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls podName:e12bca60-c1a9-46f2-9a0a-b062edfe2e7d nodeName:}" failed. No retries permitted until 2026-11-01 07:04:51.11441976 +0000 UTC m=+3306.667208933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls") pod "maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" (UID: "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d") : secret "maas-api-e2e-mt-c831a4c0-serving-cert" not found Nov 01 07:04:50.615553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.615487 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.621677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.621638 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.673681 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.673649 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkr7h\" (UniqueName: \"kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:50.694032 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.693979 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 07:04:50.827378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827341 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827379 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827397 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827424 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827531 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827566 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827596 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827663 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:50.827903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827669 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:50.827903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827686 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:50.827903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827733 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.827903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.827773 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket\") pod \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\" (UID: \"a2f48f22-54e0-4d28-a631-d5e7aef264dd\") " Nov 01 07:04:50.828222 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828136 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:50.828348 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828315 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:50.828504 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828487 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.828504 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828509 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.828671 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828523 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.828671 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828536 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.828671 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.828551 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.830717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.830692 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:04:50.830818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.830774 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:04:50.831009 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.830982 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "kube-api-access-dnn2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:50.831933 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.831906 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token" pod "a2f48f22-54e0-4d28-a631-d5e7aef264dd" (UID: "a2f48f22-54e0-4d28-a631-d5e7aef264dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:50.873365 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.873331 4914 generic.go:356] "Generic (PLEG): container finished" podID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerID="c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4" exitCode=0 Nov 01 07:04:50.873500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.873381 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" Nov 01 07:04:50.873500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.873399 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" event={"ID":"a2f48f22-54e0-4d28-a631-d5e7aef264dd","Type":"ContainerDied","Data":"c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4"} Nov 01 07:04:50.873500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.873441 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn" event={"ID":"a2f48f22-54e0-4d28-a631-d5e7aef264dd","Type":"ContainerDied","Data":"bc8ced5af420226690c80d4844e5c9cc97a4eb3066ce25e9bb55517f3aa80372"} Nov 01 07:04:50.873500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.873462 4914 scope.go:122] "RemoveContainer" containerID="c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4" Nov 01 07:04:50.891136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.891113 4914 scope.go:122] "RemoveContainer" containerID="c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4" Nov 01 07:04:50.891573 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:50.891540 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4\": container with ID starting with c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4 not found: ID does not exist" containerID="c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4" Nov 01 07:04:50.891697 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.891580 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4"} err="failed to get container status \"c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4\": rpc error: code = NotFound desc = could not find container \"c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4\": container with ID starting with c73146a48a66dd679343f23ffbe008c178d59fe04e6515605212f448a56929b4 not found: ID does not exist" Nov 01 07:04:50.936085 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.933834 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnn2s\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-kube-api-access-dnn2s\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.936085 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.933873 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.936085 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.933888 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:50.936085 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:50.933902 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f48f22-54e0-4d28-a631-d5e7aef264dd-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:51.045253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.045164 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn"] Nov 01 07:04:51.049569 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.049531 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-r9phn"] Nov 01 07:04:51.136776 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.136738 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:51.139337 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.139318 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\") pod \"maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:51.411541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.411503 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:51.785722 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.785691 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh"] Nov 01 07:04:51.787386 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:04:51.787357 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12bca60_c1a9_46f2_9a0a_b062edfe2e7d.slice/crio-6f3fa2628fc1d71bc24bdcb04b0772cf1ab25c9360da22ff4086c4c681445283 WatchSource:0}: Error finding container 6f3fa2628fc1d71bc24bdcb04b0772cf1ab25c9360da22ff4086c4c681445283: Status 404 returned error can't find the container with id 6f3fa2628fc1d71bc24bdcb04b0772cf1ab25c9360da22ff4086c4c681445283 Nov 01 07:04:51.788696 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.788680 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:04:51.881719 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:51.881681 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" event={"ID":"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d","Type":"ContainerStarted","Data":"6f3fa2628fc1d71bc24bdcb04b0772cf1ab25c9360da22ff4086c4c681445283"} Nov 01 07:04:52.618249 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:52.618216 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" path="/var/lib/kubelet/pods/a2f48f22-54e0-4d28-a631-d5e7aef264dd/volumes" Nov 01 07:04:52.892757 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:52.892672 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" event={"ID":"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d","Type":"ContainerStarted","Data":"07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77"} Nov 01 07:04:52.892757 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:52.892737 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:04:53.007685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:53.007636 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" podStartSLOduration=2.177553708 podStartE2EDuration="3.007623524s" totalImagesPullingTime="830.069816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:04:53.005840654 +0000 UTC m=+3308.558629848" watchObservedRunningTime="2026-11-01 07:04:53.007623524 +0000 UTC m=+3308.560412718" Nov 01 07:04:57.089732 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.089671 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 07:04:57.205217 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.205181 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume\") pod \"22c047da-a881-4066-ad57-73e37e66980c\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " Nov 01 07:04:57.205420 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.205257 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8\" (UniqueName: \"kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8\") pod \"22c047da-a881-4066-ad57-73e37e66980c\" (UID: \"22c047da-a881-4066-ad57-73e37e66980c\") " Nov 01 07:04:57.205606 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.205582 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume" pod "22c047da-a881-4066-ad57-73e37e66980c" (UID: "22c047da-a881-4066-ad57-73e37e66980c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:04:57.205754 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.205739 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c047da-a881-4066-ad57-73e37e66980c-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:57.207599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.207573 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8" pod "22c047da-a881-4066-ad57-73e37e66980c" (UID: "22c047da-a881-4066-ad57-73e37e66980c"). InnerVolumeSpecName "kube-api-access-chff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:04:57.307359 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.307328 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chff8\" (UniqueName: \"kubernetes.io/projected/22c047da-a881-4066-ad57-73e37e66980c-kube-api-access-chff8\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:04:57.924182 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.924151 4914 generic.go:356] "Generic (PLEG): container finished" podID="22c047da-a881-4066-ad57-73e37e66980c" containerID="acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07" exitCode=0 Nov 01 07:04:57.924368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.924199 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" Nov 01 07:04:57.924368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.924235 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" event={"ID":"22c047da-a881-4066-ad57-73e37e66980c","Type":"ContainerDied","Data":"acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07"} Nov 01 07:04:57.924368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.924306 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf" event={"ID":"22c047da-a881-4066-ad57-73e37e66980c","Type":"ContainerDied","Data":"467cf3481ac279021cea000ebe1a10b6f40ffb3ba2e71340fd046603b95980d2"} Nov 01 07:04:57.924368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.924337 4914 scope.go:122] "RemoveContainer" containerID="acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07" Nov 01 07:04:57.934431 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.934232 4914 scope.go:122] "RemoveContainer" containerID="acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07" Nov 01 07:04:57.934558 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:04:57.934541 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07\": container with ID starting with acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07 not found: ID does not exist" containerID="acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07" Nov 01 07:04:57.934621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:57.934567 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07"} err="failed to get container status \"acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07\": rpc error: code = NotFound desc = could not find container \"acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07\": container with ID starting with acd9b1bf256e0f32a9df8e3d82d72677e96a587ffbec5147c3396f3ca2956e07 not found: ID does not exist" Nov 01 07:04:58.087216 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:58.087184 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf"] Nov 01 07:04:58.091075 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:58.091049 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc6e48-6869d88cd5-lz2sf"] Nov 01 07:04:58.617266 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:58.617231 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c047da-a881-4066-ad57-73e37e66980c" path="/var/lib/kubelet/pods/22c047da-a881-4066-ad57-73e37e66980c/volumes" Nov 01 07:04:58.904723 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:04:58.904640 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:05:07.593678 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:05:07.593635 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:05:30.127774 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.127743 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt"] Nov 01 07:05:30.128373 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.128359 4914 cpu_manager.go:414] "containerMap: removing container" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" Nov 01 07:05:30.128373 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.128374 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" Nov 01 07:05:30.128463 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.128393 4914 cpu_manager.go:414] "containerMap: removing container" podUID="22c047da-a881-4066-ad57-73e37e66980c" containerName="payload-processing" Nov 01 07:05:30.128463 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.128398 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c047da-a881-4066-ad57-73e37e66980c" containerName="payload-processing" Nov 01 07:05:30.128528 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.128486 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f48f22-54e0-4d28-a631-d5e7aef264dd" containerName="istio-proxy" Nov 01 07:05:30.128528 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.128497 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c047da-a881-4066-ad57-73e37e66980c" containerName="payload-processing" Nov 01 07:05:30.131689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.131671 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.183734 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.183707 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c831a4c0-openshift-default-dockercfg-mtmts\"" Nov 01 07:05:30.236021 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.235989 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt"] Nov 01 07:05:30.271649 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.271608 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.271827 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.271700 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.271827 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.271724 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.271979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.271948 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.272019 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.272004 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.272077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.272062 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.272157 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.272143 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nr4\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.272203 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.272167 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.272203 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.272186 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.373895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.373852 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374072 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.373916 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374072 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.373944 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374072 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374059 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374090 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374143 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374199 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nr4\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374225 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374494 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374305 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374494 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374401 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374779 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374753 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.374942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374922 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.375032 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374813 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.375032 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.374945 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.376986 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.376968 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.377227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.377206 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.430006 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.429946 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.430142 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.430123 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nr4\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.445534 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.445513 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:30.871771 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.871743 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt"] Nov 01 07:05:30.873435 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:05:30.873409 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca370f6_ce4f_45f3_b8c8_bf2b8e065f1f.slice/crio-2094c395416ed9b3039092f1b7974737e932b9a7e2c678d3129d6a9a5c0fd616 WatchSource:0}: Error finding container 2094c395416ed9b3039092f1b7974737e932b9a7e2c678d3129d6a9a5c0fd616: Status 404 returned error can't find the container with id 2094c395416ed9b3039092f1b7974737e932b9a7e2c678d3129d6a9a5c0fd616 Nov 01 07:05:30.875759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.875724 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:30.875862 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.875798 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:30.875862 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:30.875844 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:31.159286 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:31.159176 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" event={"ID":"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f","Type":"ContainerStarted","Data":"f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767"} Nov 01 07:05:31.159286 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:31.159237 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" event={"ID":"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f","Type":"ContainerStarted","Data":"2094c395416ed9b3039092f1b7974737e932b9a7e2c678d3129d6a9a5c0fd616"} Nov 01 07:05:31.275067 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:31.275005 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podStartSLOduration=1.274985139 podStartE2EDuration="1.274985139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:31.273635554 +0000 UTC m=+3346.826424774" watchObservedRunningTime="2026-11-01 07:05:31.274985139 +0000 UTC m=+3346.827774335" Nov 01 07:05:31.464374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:31.450913 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:31.464374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:31.451039 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 07:05:31.464374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:31.451081 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 07:05:32.466301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:32.450605 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 07:05:32.466301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:32.450671 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 07:05:33.460298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:33.450438 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 07:05:33.460298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:33.450527 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 07:05:34.468298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:34.452595 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 07:05:34.468298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:34.452657 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 07:05:35.456311 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:35.455122 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 07:05:35.456311 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:35.455184 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 07:05:36.453302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:36.451666 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 01 07:05:36.453302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:36.451727 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 01 07:05:37.450730 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:37.450696 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:37.451134 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:37.451115 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:37.451792 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:37.451774 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:05:37.594200 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:05:37.594166 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:05:45.250501 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.250466 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22"] Nov 01 07:05:45.255580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.255562 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.288860 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.288821 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289030 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.288887 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289030 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.288998 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznzh\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.289051 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.289073 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.289099 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289241 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.289201 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289241 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.289231 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.289369 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.289259 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.300383 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.300351 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk"] Nov 01 07:05:45.304408 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.304390 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.355405 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.355379 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22"] Nov 01 07:05:45.390492 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390459 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.390621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390501 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.390621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390523 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.390693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390633 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.390693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390667 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.390762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390692 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.390762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390717 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.390762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390754 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.390921 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390876 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.390976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390948 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.390998 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.391225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391037 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznzh\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391036 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391168 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.391225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391221 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391264 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391326 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.391513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391369 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391396 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfxs\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.391513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391429 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.391708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391617 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.391747 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391701 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.392013 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.391990 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.393481 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.393457 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.393745 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.393715 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.408086 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.408065 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk"] Nov 01 07:05:45.447331 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.447300 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.447441 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.447405 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznzh\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.492912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.492881 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493049 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.492971 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493049 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493033 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493115 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493159 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493207 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfxs\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493241 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493300 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493501 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493322 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493501 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493338 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493640 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493698 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.493875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.493848 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.494051 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.494022 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.495767 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.495746 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.496445 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.496428 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.550244 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.550190 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.550388 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.550258 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfxs\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs\") pod \"e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.569326 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.569304 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:45.616986 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.616953 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:45.995110 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:05:45.995085 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa800fd_cf83_4ea9_83f9_5130cae57a71.slice/crio-3c036a636e35e430e9d4f3166e26465352d55f23713c3642e4fcc114429e17b4 WatchSource:0}: Error finding container 3c036a636e35e430e9d4f3166e26465352d55f23713c3642e4fcc114429e17b4: Status 404 returned error can't find the container with id 3c036a636e35e430e9d4f3166e26465352d55f23713c3642e4fcc114429e17b4 Nov 01 07:05:45.995205 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.995188 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22"] Nov 01 07:05:45.997755 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.997718 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:45.997871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.997799 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:45.997871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:45.997847 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:46.045199 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.045177 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk"] Nov 01 07:05:46.045933 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:05:46.045915 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9baa9b_e8f2_40f1_9b3c_780763f26864.slice/crio-cba62f779af7c64a42a4b893ac64fd966c57b597e8b3ba594a54139876d29699 WatchSource:0}: Error finding container cba62f779af7c64a42a4b893ac64fd966c57b597e8b3ba594a54139876d29699: Status 404 returned error can't find the container with id cba62f779af7c64a42a4b893ac64fd966c57b597e8b3ba594a54139876d29699 Nov 01 07:05:46.048762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.048734 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:46.048851 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.048798 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:46.048851 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.048825 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:05:46.291259 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.291166 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" event={"ID":"6f9baa9b-e8f2-40f1-9b3c-780763f26864","Type":"ContainerStarted","Data":"fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a"} Nov 01 07:05:46.291259 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.291227 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" event={"ID":"6f9baa9b-e8f2-40f1-9b3c-780763f26864","Type":"ContainerStarted","Data":"cba62f779af7c64a42a4b893ac64fd966c57b597e8b3ba594a54139876d29699"} Nov 01 07:05:46.292986 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.292959 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" event={"ID":"7aa800fd-cf83-4ea9-83f9-5130cae57a71","Type":"ContainerStarted","Data":"97b66d4a700c2618351709a24d35ed7dff6937b2a4b5579455acbe46ff1b5622"} Nov 01 07:05:46.293121 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.293006 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" event={"ID":"7aa800fd-cf83-4ea9-83f9-5130cae57a71","Type":"ContainerStarted","Data":"3c036a636e35e430e9d4f3166e26465352d55f23713c3642e4fcc114429e17b4"} Nov 01 07:05:46.408843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.408770 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podStartSLOduration=1.408750008 podStartE2EDuration="1.408750008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:46.404991665 +0000 UTC m=+3361.957780860" watchObservedRunningTime="2026-11-01 07:05:46.408750008 +0000 UTC m=+3361.961539203" Nov 01 07:05:46.532266 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.532203 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podStartSLOduration=1.5321884369999998 podStartE2EDuration="1.532188437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:05:46.528886992 +0000 UTC m=+3362.081676188" watchObservedRunningTime="2026-11-01 07:05:46.532188437 +0000 UTC m=+3362.084977630" Nov 01 07:05:46.572357 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.570396 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:46.625801 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:46.625768 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:47.576301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:47.575384 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 01 07:05:47.576301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:47.575443 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 01 07:05:47.618386 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:47.618348 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:47.618558 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:47.618404 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 07:05:48.571300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:48.570715 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 01 07:05:48.571300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:48.570783 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 01 07:05:48.617583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:48.617485 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:48.617583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:48.617532 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 07:05:49.572299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:49.571647 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 01 07:05:49.572299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:49.571720 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 01 07:05:49.618064 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:49.618029 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:49.618453 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:49.618078 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 07:05:50.579676 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:50.579638 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 01 07:05:50.579987 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:50.579957 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 01 07:05:50.617662 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:50.617634 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:50.617821 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:50.617681 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 07:05:51.586298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:51.585829 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 01 07:05:51.586298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:51.585903 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 01 07:05:51.618313 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:51.618261 4914 patch_prober.go:29] interesting pod/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 01 07:05:51.618455 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:51.618325 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 01 07:05:52.627302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:52.626579 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:52.628183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:52.628159 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:52.629227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:52.629203 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:05:52.676171 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:52.676139 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:52.676603 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:52.676576 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:05:52.677731 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:05:52.677709 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:06:07.592952 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:07.592859 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:06:07.686801 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.686764 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-39603a"] Nov 01 07:06:07.692223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.692199 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:07.746984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.746951 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-39603a"] Nov 01 07:06:07.747163 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.747143 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-557zr\"" Nov 01 07:06:07.807860 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.807812 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvfk\" (UniqueName: \"kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk\") pod \"test-curl-7695-39603a\" (UID: \"4f7ca86d-957e-4a2b-83dd-69534be70540\") " pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:07.908533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.908501 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvfk\" (UniqueName: \"kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk\") pod \"test-curl-7695-39603a\" (UID: \"4f7ca86d-957e-4a2b-83dd-69534be70540\") " pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:07.964598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:07.964568 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvfk\" (UniqueName: \"kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk\") pod \"test-curl-7695-39603a\" (UID: \"4f7ca86d-957e-4a2b-83dd-69534be70540\") " pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:08.005537 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:08.005505 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:08.385985 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:08.385961 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-39603a"] Nov 01 07:06:08.388114 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:06:08.388084 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f7ca86d_957e_4a2b_83dd_69534be70540.slice/crio-6727394ae712b8ade27a3b305ea09dc0e9724c2d819241a37c0d2d8e8c143576 WatchSource:0}: Error finding container 6727394ae712b8ade27a3b305ea09dc0e9724c2d819241a37c0d2d8e8c143576: Status 404 returned error can't find the container with id 6727394ae712b8ade27a3b305ea09dc0e9724c2d819241a37c0d2d8e8c143576 Nov 01 07:06:08.470342 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:08.470311 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-39603a" event={"ID":"4f7ca86d-957e-4a2b-83dd-69534be70540","Type":"ContainerStarted","Data":"cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195"} Nov 01 07:06:08.470459 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:08.470360 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-39603a" event={"ID":"4f7ca86d-957e-4a2b-83dd-69534be70540","Type":"ContainerStarted","Data":"6727394ae712b8ade27a3b305ea09dc0e9724c2d819241a37c0d2d8e8c143576"} Nov 01 07:06:08.581188 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:08.581129 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7695-39603a" podStartSLOduration=1.581114403 podStartE2EDuration="1.581114403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:08.580901873 +0000 UTC m=+3384.133691067" watchObservedRunningTime="2026-11-01 07:06:08.581114403 +0000 UTC m=+3384.133903596" Nov 01 07:06:09.982493 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:09.982462 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-39603a"] Nov 01 07:06:09.987398 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:09.987367 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-39603a"] Nov 01 07:06:10.482914 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:10.482875 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7695-39603a" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" containerName="test-curl-7695-39603a" containerID="cri-o://cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195" gracePeriod=2 Nov 01 07:06:10.533827 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:10.533794 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-39603a\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:12.728146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:12.728121 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:12.868268 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:12.868236 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk\" (UniqueName: \"kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk\") pod \"4f7ca86d-957e-4a2b-83dd-69534be70540\" (UID: \"4f7ca86d-957e-4a2b-83dd-69534be70540\") " Nov 01 07:06:12.870479 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:12.870456 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk" pod "4f7ca86d-957e-4a2b-83dd-69534be70540" (UID: "4f7ca86d-957e-4a2b-83dd-69534be70540"). InnerVolumeSpecName "kube-api-access-fcvfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:06:12.970216 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:12.970179 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvfk\" (UniqueName: \"kubernetes.io/projected/4f7ca86d-957e-4a2b-83dd-69534be70540-kube-api-access-fcvfk\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:06:13.505499 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:13.505464 4914 generic.go:356] "Generic (PLEG): container finished" podID="4f7ca86d-957e-4a2b-83dd-69534be70540" containerID="cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195" exitCode=137 Nov 01 07:06:13.505697 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:13.505522 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:13.505697 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:13.505564 4914 scope.go:122] "RemoveContainer" containerID="cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195" Nov 01 07:06:13.515575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:13.515558 4914 scope.go:122] "RemoveContainer" containerID="cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195" Nov 01 07:06:13.515854 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:13.515836 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195\": container with ID starting with cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195 not found: ID does not exist" containerID="cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195" Nov 01 07:06:13.515913 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:13.515863 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195"} err="failed to get container status \"cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195\": rpc error: code = NotFound desc = could not find container \"cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195\": container with ID starting with cb54f30ce55e1b05f83f451cbcd7053bb74cff39c9140eceab3e2931cfccd195 not found: ID does not exist" Nov 01 07:06:13.565009 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:13.564975 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-39603a\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:14.618350 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:14.618317 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" path="/var/lib/kubelet/pods/4f7ca86d-957e-4a2b-83dd-69534be70540/volumes" Nov 01 07:06:14.664460 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:14.664426 4914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7695-39603a\" is forbidden: User \"system:node:ip-10-0-130-54.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-54.us-west-1.compute.internal' and this object" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" pod="opendatahub/test-curl-7695-39603a" Nov 01 07:06:35.841508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.841476 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj"] Nov 01 07:06:35.844027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.842091 4914 cpu_manager.go:414] "containerMap: removing container" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" containerName="test-curl-7695-39603a" Nov 01 07:06:35.844027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.842105 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" containerName="test-curl-7695-39603a" Nov 01 07:06:35.844027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.842213 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7ca86d-957e-4a2b-83dd-69534be70540" containerName="test-curl-7695-39603a" Nov 01 07:06:35.845573 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.845557 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:35.905134 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.905106 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:35.905292 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.905146 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:35.905292 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.905176 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:35.905292 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.905238 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ckp\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:35.951449 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:35.951423 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj"] Nov 01 07:06:36.006725 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.006696 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.006725 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.006734 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.006943 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.006755 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.007083 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.007058 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ckp\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.007519 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.007492 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.007657 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.007638 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.009614 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.009592 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.063325 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.063297 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ckp\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.158847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.158820 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.534127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.534087 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj"] Nov 01 07:06:36.535586 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:06:36.535556 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8646c50c_4064_452a_9897_a5e46448ea7c.slice/crio-5df2e96d6055515f6380c09dcf528fc091aa9a7f553a731ec9683452a36b28c1 WatchSource:0}: Error finding container 5df2e96d6055515f6380c09dcf528fc091aa9a7f553a731ec9683452a36b28c1: Status 404 returned error can't find the container with id 5df2e96d6055515f6380c09dcf528fc091aa9a7f553a731ec9683452a36b28c1 Nov 01 07:06:36.710727 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.710693 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" event={"ID":"8646c50c-4064-452a-9897-a5e46448ea7c","Type":"ContainerStarted","Data":"fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9"} Nov 01 07:06:36.710932 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.710746 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" event={"ID":"8646c50c-4064-452a-9897-a5e46448ea7c","Type":"ContainerStarted","Data":"5df2e96d6055515f6380c09dcf528fc091aa9a7f553a731ec9683452a36b28c1"} Nov 01 07:06:36.710932 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.710767 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:06:36.825650 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:36.825554 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" podStartSLOduration=1.825542349 podStartE2EDuration="1.825542349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:06:36.822741846 +0000 UTC m=+3412.375531039" watchObservedRunningTime="2026-11-01 07:06:36.825542349 +0000 UTC m=+3412.378331586" Nov 01 07:06:37.593449 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:37.593415 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:06:47.719708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:47.719660 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:06:57.717692 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:57.717656 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:06:59.567397 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.567359 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh"] Nov 01 07:06:59.567769 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.567622 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" podUID="e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" containerName="maas-api" containerID="cri-o://07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77" gracePeriod=30 Nov 01 07:06:59.866027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.866000 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:06:59.883812 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.883780 4914 generic.go:356] "Generic (PLEG): container finished" podID="e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" containerID="07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77" exitCode=0 Nov 01 07:06:59.883960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.883867 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" event={"ID":"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d","Type":"ContainerDied","Data":"07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77"} Nov 01 07:06:59.883960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.883877 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" Nov 01 07:06:59.883960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.883903 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh" event={"ID":"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d","Type":"ContainerDied","Data":"6f3fa2628fc1d71bc24bdcb04b0772cf1ab25c9360da22ff4086c4c681445283"} Nov 01 07:06:59.883960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.883924 4914 scope.go:122] "RemoveContainer" containerID="07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77" Nov 01 07:06:59.894051 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.894030 4914 scope.go:122] "RemoveContainer" containerID="07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77" Nov 01 07:06:59.894353 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:06:59.894327 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77\": container with ID starting with 07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77 not found: ID does not exist" containerID="07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77" Nov 01 07:06:59.894447 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.894359 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77"} err="failed to get container status \"07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77\": rpc error: code = NotFound desc = could not find container \"07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77\": container with ID starting with 07db3423ecec5b631e924313498650c6ad7ceb3f21182e8cfe7cc0aaaccedb77 not found: ID does not exist" Nov 01 07:06:59.945734 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.945710 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h\" (UniqueName: \"kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h\") pod \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " Nov 01 07:06:59.945852 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.945800 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\") pod \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " Nov 01 07:06:59.945852 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.945842 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca\") pod \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " Nov 01 07:06:59.945962 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.945947 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs\") pod \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\" (UID: \"e12bca60-c1a9-46f2-9a0a-b062edfe2e7d\") " Nov 01 07:06:59.946288 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.946237 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca" pod "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" (UID: "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:06:59.946583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.946396 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:06:59.948096 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.948072 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h" pod "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" (UID: "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d"). InnerVolumeSpecName "kube-api-access-rkr7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:06:59.948189 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.948168 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs" pod "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" (UID: "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:06:59.948189 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:06:59.948179 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls" pod "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" (UID: "e12bca60-c1a9-46f2-9a0a-b062edfe2e7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:07:00.047394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:00.047364 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:00.047394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:00.047389 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:00.047394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:00.047399 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkr7h\" (UniqueName: \"kubernetes.io/projected/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d-kube-api-access-rkr7h\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:00.310824 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:00.310783 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh"] Nov 01 07:07:00.314488 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:00.314398 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c831a4c0-7bbb8bb5d4-29brh"] Nov 01 07:07:00.648046 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:00.648000 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" path="/var/lib/kubelet/pods/e12bca60-c1a9-46f2-9a0a-b062edfe2e7d/volumes" Nov 01 07:07:01.144267 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.144236 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj"] Nov 01 07:07:01.145970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.145943 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:01.177053 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.177024 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4"] Nov 01 07:07:01.177652 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.177639 4914 cpu_manager.go:414] "containerMap: removing container" podUID="e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" containerName="maas-api" Nov 01 07:07:01.177711 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.177654 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" containerName="maas-api" Nov 01 07:07:01.177773 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.177764 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12bca60-c1a9-46f2-9a0a-b062edfe2e7d" containerName="maas-api" Nov 01 07:07:01.182397 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.182381 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.236463 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.236443 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4"] Nov 01 07:07:01.278465 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.278432 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.278629 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.278526 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.278629 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.278604 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fnd\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.278884 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.278647 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.379390 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.379355 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.379558 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.379420 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.379558 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.379479 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fnd\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.379558 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.379532 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.379734 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:01.379716 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:01.379770 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:01.379751 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:01.379834 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:01.379823 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials podName:6fce4c4f-d1c8-4592-96b8-ebf679bf2af1 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:01.879801948 +0000 UTC m=+3437.432591122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1") : secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:01.380141 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.380120 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.380180 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.380159 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.436661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.436597 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fnd\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.885663 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:01.885630 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:01.886140 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:01.885800 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:01.886140 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:01.885831 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:01.886140 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:01.885898 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials podName:6fce4c4f-d1c8-4592-96b8-ebf679bf2af1 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:02.88587831 +0000 UTC m=+3438.438667482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1") : secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:02.895750 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:02.895715 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:02.896118 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:02.895880 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:02.896118 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:02.895910 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:02.896118 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:02.895977 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials podName:6fce4c4f-d1c8-4592-96b8-ebf679bf2af1 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:04.89595933 +0000 UTC m=+3440.448748508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1") : secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:04.262210 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:04.262174 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj"] Nov 01 07:07:04.373163 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:04.373130 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb"] Nov 01 07:07:04.955039 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:04.955005 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:04.955259 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:04.955159 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:04.955259 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:04.955188 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:04.955259 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:04.955255 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials podName:6fce4c4f-d1c8-4592-96b8-ebf679bf2af1 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:08.955239741 +0000 UTC m=+3444.508028916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1") : secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:06.144725 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.144686 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" containerID="cri-o://fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9" gracePeriod=25 Nov 01 07:07:06.408100 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.408041 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:07:06.578764 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.578727 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials\") pod \"8646c50c-4064-452a-9897-a5e46448ea7c\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " Nov 01 07:07:06.578938 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.578880 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume\") pod \"8646c50c-4064-452a-9897-a5e46448ea7c\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " Nov 01 07:07:06.579006 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.578991 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume\") pod \"8646c50c-4064-452a-9897-a5e46448ea7c\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " Nov 01 07:07:06.579070 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.579057 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp\") pod \"8646c50c-4064-452a-9897-a5e46448ea7c\" (UID: \"8646c50c-4064-452a-9897-a5e46448ea7c\") " Nov 01 07:07:06.579330 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.579266 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume" pod "8646c50c-4064-452a-9897-a5e46448ea7c" (UID: "8646c50c-4064-452a-9897-a5e46448ea7c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:06.579428 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.579393 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume" pod "8646c50c-4064-452a-9897-a5e46448ea7c" (UID: "8646c50c-4064-452a-9897-a5e46448ea7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:06.579508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.579494 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:06.579609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.579512 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8646c50c-4064-452a-9897-a5e46448ea7c-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:06.580945 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.580926 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials" pod "8646c50c-4064-452a-9897-a5e46448ea7c" (UID: "8646c50c-4064-452a-9897-a5e46448ea7c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:06.581531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.581506 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp" pod "8646c50c-4064-452a-9897-a5e46448ea7c" (UID: "8646c50c-4064-452a-9897-a5e46448ea7c"). InnerVolumeSpecName "kube-api-access-k9ckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:06.680648 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.680560 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:06.680648 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.680595 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ckp\" (UniqueName: \"kubernetes.io/projected/8646c50c-4064-452a-9897-a5e46448ea7c-kube-api-access-k9ckp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:06.932781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.932689 4914 generic.go:356] "Generic (PLEG): container finished" podID="8646c50c-4064-452a-9897-a5e46448ea7c" containerID="fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9" exitCode=0 Nov 01 07:07:06.932781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.932734 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" Nov 01 07:07:06.932781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.932768 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" event={"ID":"8646c50c-4064-452a-9897-a5e46448ea7c","Type":"ContainerDied","Data":"fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9"} Nov 01 07:07:06.933069 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.932828 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj" event={"ID":"8646c50c-4064-452a-9897-a5e46448ea7c","Type":"ContainerDied","Data":"5df2e96d6055515f6380c09dcf528fc091aa9a7f553a731ec9683452a36b28c1"} Nov 01 07:07:06.933069 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.932850 4914 scope.go:122] "RemoveContainer" containerID="fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9" Nov 01 07:07:06.942374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.942348 4914 scope.go:122] "RemoveContainer" containerID="fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9" Nov 01 07:07:06.942661 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:06.942630 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9\": container with ID starting with fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9 not found: ID does not exist" containerID="fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9" Nov 01 07:07:06.942753 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:06.942665 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9"} err="failed to get container status \"fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9\": rpc error: code = NotFound desc = could not find container \"fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9\": container with ID starting with fba32acaf735427b24caf76468ddc1da85b37c1f5e051d82467995e9bb9889d9 not found: ID does not exist" Nov 01 07:07:07.095192 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.095165 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj"] Nov 01 07:07:07.098798 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.098778 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-555dzqdnj"] Nov 01 07:07:07.592358 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:07.592325 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:07:07.776075 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.776040 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4"] Nov 01 07:07:07.776435 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:07.776409 4914 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" podUID="6fce4c4f-d1c8-4592-96b8-ebf679bf2af1" Nov 01 07:07:07.806922 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.806897 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw"] Nov 01 07:07:07.807619 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.807599 4914 cpu_manager.go:414] "containerMap: removing container" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" Nov 01 07:07:07.807619 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.807620 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" Nov 01 07:07:07.807824 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.807807 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" containerName="payload-processing" Nov 01 07:07:07.812736 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.812717 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.887269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.887242 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw"] Nov 01 07:07:07.894561 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.894536 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk8v7\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.894733 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.894714 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.894891 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.894877 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.894997 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.894985 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.940248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.940222 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:07.947604 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.947582 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:07.996372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.996345 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.996490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.996442 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.996560 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.996486 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.996681 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.996666 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk8v7\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.997119 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.997099 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.997235 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.997217 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:07.999166 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:07.999147 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:08.056059 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.056039 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk8v7\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:08.097639 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.097617 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume\") pod \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " Nov 01 07:07:08.097766 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.097661 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd\") pod \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " Nov 01 07:07:08.097832 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.097782 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume\") pod \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " Nov 01 07:07:08.098095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.098070 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume" pod "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:08.098166 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.098102 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume" pod "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:08.098237 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.098223 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:08.098306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.098240 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:08.099943 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.099922 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd" pod "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1"). InnerVolumeSpecName "kube-api-access-l7fnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:08.126716 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.126697 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:08.288600 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.241801 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7fnd\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-kube-api-access-l7fnd\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:08.563536 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.563513 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw"] Nov 01 07:07:08.564415 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:07:08.564390 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b36c584_8644_4788_a7b2_1eedef0a51b3.slice/crio-f55f25939acce9e13b0d1e198454b26e875c6c20468aeb71ba8ee11158396949 WatchSource:0}: Error finding container f55f25939acce9e13b0d1e198454b26e875c6c20468aeb71ba8ee11158396949: Status 404 returned error can't find the container with id f55f25939acce9e13b0d1e198454b26e875c6c20468aeb71ba8ee11158396949 Nov 01 07:07:08.617083 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.617061 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8646c50c-4064-452a-9897-a5e46448ea7c" path="/var/lib/kubelet/pods/8646c50c-4064-452a-9897-a5e46448ea7c/volumes" Nov 01 07:07:08.949555 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.949516 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" event={"ID":"6b36c584-8644-4788-a7b2-1eedef0a51b3","Type":"ContainerStarted","Data":"8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514"} Nov 01 07:07:08.949742 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.949584 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:08.949742 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.949585 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" event={"ID":"6b36c584-8644-4788-a7b2-1eedef0a51b3","Type":"ContainerStarted","Data":"f55f25939acce9e13b0d1e198454b26e875c6c20468aeb71ba8ee11158396949"} Nov 01 07:07:08.949742 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:08.949719 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:09.049242 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.049204 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4\" (UID: \"6fce4c4f-d1c8-4592-96b8-ebf679bf2af1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" Nov 01 07:07:09.050243 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.049614 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:09.051532 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.051517 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:09.051696 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.051677 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials podName:6fce4c4f-d1c8-4592-96b8-ebf679bf2af1 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:17.051661145 +0000 UTC m=+3452.604450320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4" (UID: "6fce4c4f-d1c8-4592-96b8-ebf679bf2af1") : secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:09.121562 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.121503 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" podStartSLOduration=2.121489017 podStartE2EDuration="2.121489017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:09.12053098 +0000 UTC m=+3444.673320175" watchObservedRunningTime="2026-11-01 07:07:09.121489017 +0000 UTC m=+3444.674278248" Nov 01 07:07:09.263407 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.263325 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerName="payload-processing" containerID="cri-o://2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07" gracePeriod=25 Nov 01 07:07:09.310234 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.310203 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4"] Nov 01 07:07:09.314456 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.314436 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-7977k2qz4"] Nov 01 07:07:09.353235 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.353211 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:09.374622 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.374590 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" containerName="payload-pre-processing" containerID="cri-o://8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d" gracePeriod=25 Nov 01 07:07:09.663298 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.662816 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:09.663298 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.662852 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-6f862af0425e: secret "agc-cred-6f862af0425e" not found Nov 01 07:07:09.663298 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.662873 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-ea026b0e6c64: secret "agc-cred-ea026b0e6c64" not found Nov 01 07:07:09.663298 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.662903 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b: [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:09.663298 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:09.662964 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials podName:c00a77b6-a06b-42a7-b825-c6cb64b40551 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:10.162944132 +0000 UTC m=+3445.715733322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551") : [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:09.749115 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.748834 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw"] Nov 01 07:07:09.749115 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:09.749068 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" containerID="cri-o://ac9f56462feb89dea1b1ce302ee914062820df3924324bf151007559c7068acf" gracePeriod=30 Nov 01 07:07:10.183570 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:10.183539 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:10.183570 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:10.183563 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-6f862af0425e: secret "agc-cred-6f862af0425e" not found Nov 01 07:07:10.183570 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:10.183575 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-ea026b0e6c64: secret "agc-cred-ea026b0e6c64" not found Nov 01 07:07:10.183799 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:10.183586 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b: [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:10.183799 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:10.183637 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials podName:c00a77b6-a06b-42a7-b825-c6cb64b40551 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:11.183622626 +0000 UTC m=+3446.736411798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551") : [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:10.616640 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:10.616610 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fce4c4f-d1c8-4592-96b8-ebf679bf2af1" path="/var/lib/kubelet/pods/6fce4c4f-d1c8-4592-96b8-ebf679bf2af1/volumes" Nov 01 07:07:11.193839 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:11.193806 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:11.193839 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:11.193833 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-6f862af0425e: secret "agc-cred-6f862af0425e" not found Nov 01 07:07:11.193839 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:11.193842 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-ea026b0e6c64: secret "agc-cred-ea026b0e6c64" not found Nov 01 07:07:11.194505 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:11.193853 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b: [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:11.194505 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:11.193908 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials podName:c00a77b6-a06b-42a7-b825-c6cb64b40551 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:13.193894779 +0000 UTC m=+3448.746683951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551") : [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:11.525018 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:11.524945 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt"] Nov 01 07:07:11.525214 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:11.525178 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" containerID="cri-o://f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767" gracePeriod=30 Nov 01 07:07:11.537825 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:11.537794 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22"] Nov 01 07:07:11.538074 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:11.538031 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" containerID="cri-o://97b66d4a700c2618351709a24d35ed7dff6937b2a4b5579455acbe46ff1b5622" gracePeriod=30 Nov 01 07:07:11.561397 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:11.561374 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk"] Nov 01 07:07:11.561728 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:11.561572 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" containerID="cri-o://fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a" gracePeriod=30 Nov 01 07:07:13.215779 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:13.215748 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:13.215779 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:13.215782 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-6f862af0425e: secret "agc-cred-6f862af0425e" not found Nov 01 07:07:13.216224 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:13.215796 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-ea026b0e6c64: secret "agc-cred-ea026b0e6c64" not found Nov 01 07:07:13.216224 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:13.215806 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b: [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:13.216224 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:13.215853 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials podName:c00a77b6-a06b-42a7-b825-c6cb64b40551 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:17.215840264 +0000 UTC m=+3452.768629439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551") : [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:13.356853 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.356821 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9"] Nov 01 07:07:13.362264 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.362246 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.416827 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.416784 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.416962 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.416840 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.416962 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.416894 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.417049 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.416982 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.417049 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.417016 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.417129 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.417108 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.417256 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.417223 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.417417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.417313 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.417417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.417355 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.469186 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.469124 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9"] Nov 01 07:07:13.518344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518322 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518476 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518352 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518476 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518400 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518476 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518434 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518506 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518572 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518603 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518813 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518662 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518813 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518725 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.518936 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.518893 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.519034 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.519010 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.519093 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.519050 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.519093 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.519070 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.519396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.519138 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.520944 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.520921 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.521086 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.521068 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.574559 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.574534 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.574693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.574672 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5\") pod \"e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:13.676203 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:13.676177 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:14.103802 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.103779 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9"] Nov 01 07:07:14.104479 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:07:14.104454 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e899976_d941_49b6_8c45_fdb3998f9d1b.slice/crio-dcaafaf21f8063c1e4b13fbbf6983bb4983fea7d7928a4f38e0ce644d525e98d WatchSource:0}: Error finding container dcaafaf21f8063c1e4b13fbbf6983bb4983fea7d7928a4f38e0ce644d525e98d: Status 404 returned error can't find the container with id dcaafaf21f8063c1e4b13fbbf6983bb4983fea7d7928a4f38e0ce644d525e98d Nov 01 07:07:14.106779 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.106749 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:07:14.106846 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.106812 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:07:14.106846 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.106841 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:07:14.990307 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.990251 4914 generic.go:356] "Generic (PLEG): container finished" podID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerID="ac9f56462feb89dea1b1ce302ee914062820df3924324bf151007559c7068acf" exitCode=0 Nov 01 07:07:14.992482 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.990308 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" event={"ID":"b3460fb3-9719-4dc5-9aa9-b7e23230c69a","Type":"ContainerDied","Data":"ac9f56462feb89dea1b1ce302ee914062820df3924324bf151007559c7068acf"} Nov 01 07:07:14.992482 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.990387 4914 scope.go:122] "RemoveContainer" containerID="bb0e463cbd8e665b530aab62ed0393f77c27bd30ad2b7324cc1f966015efb3a4" Nov 01 07:07:14.994025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.993988 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" event={"ID":"5e899976-d941-49b6-8c45-fdb3998f9d1b","Type":"ContainerStarted","Data":"8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4"} Nov 01 07:07:14.994153 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:14.994046 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" event={"ID":"5e899976-d941-49b6-8c45-fdb3998f9d1b","Type":"ContainerStarted","Data":"dcaafaf21f8063c1e4b13fbbf6983bb4983fea7d7928a4f38e0ce644d525e98d"} Nov 01 07:07:15.025180 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.025116 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 07:07:15.140183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140140 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140197 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140234 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140321 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140358 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140663 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140480 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140663 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140537 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140663 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140593 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.140806 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140690 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j\") pod \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\" (UID: \"b3460fb3-9719-4dc5-9aa9-b7e23230c69a\") " Nov 01 07:07:15.141157 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140866 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:15.141157 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.140962 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:15.141322 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.141215 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:15.141390 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.141333 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.141390 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.141356 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.141390 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.141371 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.141545 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.141514 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:15.141664 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.141608 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:15.143724 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.143686 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:15.143859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.143729 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:15.143973 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.143960 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "kube-api-access-vvr5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:15.144796 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.144771 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo" pod "b3460fb3-9719-4dc5-9aa9-b7e23230c69a" (UID: "b3460fb3-9719-4dc5-9aa9-b7e23230c69a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:15.226832 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.226777 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podStartSLOduration=2.226762757 podStartE2EDuration="2.226762757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:15.175863861 +0000 UTC m=+3450.728653059" watchObservedRunningTime="2026-11-01 07:07:15.226762757 +0000 UTC m=+3450.779551950" Nov 01 07:07:15.242602 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.242575 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvr5j\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-kube-api-access-vvr5j\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.242602 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.242600 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.242761 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.242612 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.242761 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.242621 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.242761 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.242629 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.242761 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.242637 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3460fb3-9719-4dc5-9aa9-b7e23230c69a-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:15.676717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.676684 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:15.678469 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.678443 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:07:15.678607 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:15.678489 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 07:07:16.002065 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.001973 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" Nov 01 07:07:16.002065 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.002009 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw" event={"ID":"b3460fb3-9719-4dc5-9aa9-b7e23230c69a","Type":"ContainerDied","Data":"20b20b1a5a4c758947726a000162eef3babe9edc81b64e7e3afc196589772bbb"} Nov 01 07:07:16.002645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.002103 4914 scope.go:122] "RemoveContainer" containerID="ac9f56462feb89dea1b1ce302ee914062820df3924324bf151007559c7068acf" Nov 01 07:07:16.071964 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.071916 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:16.072128 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.071922 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:16.233829 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.233795 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw"] Nov 01 07:07:16.237457 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.237426 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-25719d-openshift-default-77fc995d65-h5crw"] Nov 01 07:07:16.385202 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.385173 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw"] Nov 01 07:07:16.388359 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.388327 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" podUID="6b36c584-8644-4788-a7b2-1eedef0a51b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:16.412608 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.412580 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857"] Nov 01 07:07:16.413333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.413317 4914 cpu_manager.go:414] "containerMap: removing container" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" Nov 01 07:07:16.413386 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.413334 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" Nov 01 07:07:16.413386 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.413383 4914 cpu_manager.go:414] "containerMap: removing container" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" Nov 01 07:07:16.413467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.413389 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" Nov 01 07:07:16.413507 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.413498 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" Nov 01 07:07:16.413545 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.413514 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" containerName="istio-proxy" Nov 01 07:07:16.418634 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.418609 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.474362 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.474334 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857"] Nov 01 07:07:16.481137 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.481110 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.481296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.481164 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.481354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.481261 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxlz\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.481472 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.481452 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.582238 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.582204 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.582419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.582257 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.582419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.582304 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxlz\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.582536 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.582426 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.583111 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.583084 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.583161 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.583096 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.584790 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.584770 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.617978 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.617954 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3460fb3-9719-4dc5-9aa9-b7e23230c69a" path="/var/lib/kubelet/pods/b3460fb3-9719-4dc5-9aa9-b7e23230c69a/volumes" Nov 01 07:07:16.638251 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.638225 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxlz\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz\") pod \"payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.678150 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.677263 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:07:16.678150 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.677347 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 07:07:16.732529 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.731896 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:16.892008 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:16.891736 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.026591 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.026645 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.026675 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.026722 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.026803 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.026914 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.027029 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.027108 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.027170 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert\") pod \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\" (UID: \"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f\") " Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.028418 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.029894 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.030411 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.030713 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.031553 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.030987 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.037897 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.037111 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "kube-api-access-f7nr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:17.037897 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.037248 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.037897 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.037520 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:17.041375 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.041345 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token" pod "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" (UID: "dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:17.077303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.073492 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:07:17.079641 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.079611 4914 generic.go:356] "Generic (PLEG): container finished" podID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerID="fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a" exitCode=0 Nov 01 07:07:17.079791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.079732 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" event={"ID":"6f9baa9b-e8f2-40f1-9b3c-780763f26864","Type":"ContainerDied","Data":"fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a"} Nov 01 07:07:17.079791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.079784 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" event={"ID":"6f9baa9b-e8f2-40f1-9b3c-780763f26864","Type":"ContainerDied","Data":"cba62f779af7c64a42a4b893ac64fd966c57b597e8b3ba594a54139876d29699"} Nov 01 07:07:17.079924 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.079819 4914 scope.go:122] "RemoveContainer" containerID="fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a" Nov 01 07:07:17.083880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.083854 4914 generic.go:356] "Generic (PLEG): container finished" podID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerID="97b66d4a700c2618351709a24d35ed7dff6937b2a4b5579455acbe46ff1b5622" exitCode=0 Nov 01 07:07:17.083976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.083918 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" event={"ID":"7aa800fd-cf83-4ea9-83f9-5130cae57a71","Type":"ContainerDied","Data":"97b66d4a700c2618351709a24d35ed7dff6937b2a4b5579455acbe46ff1b5622"} Nov 01 07:07:17.085391 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.085369 4914 generic.go:356] "Generic (PLEG): container finished" podID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerID="f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767" exitCode=0 Nov 01 07:07:17.085721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.085708 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" Nov 01 07:07:17.086361 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.086339 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" event={"ID":"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f","Type":"ContainerDied","Data":"f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767"} Nov 01 07:07:17.086450 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.086411 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt" event={"ID":"dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f","Type":"ContainerDied","Data":"2094c395416ed9b3039092f1b7974737e932b9a7e2c678d3129d6a9a5c0fd616"} Nov 01 07:07:17.089559 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.089542 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:07:17.101689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.101668 4914 scope.go:122] "RemoveContainer" containerID="fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a" Nov 01 07:07:17.102012 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.101981 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a\": container with ID starting with fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a not found: ID does not exist" containerID="fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a" Nov 01 07:07:17.102105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.102021 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a"} err="failed to get container status \"fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a\": rpc error: code = NotFound desc = could not find container \"fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a\": container with ID starting with fab8f804ab26b3cc58372fe3d3293f757038801dfd1e10e0b242272d2110cb9a not found: ID does not exist" Nov 01 07:07:17.102105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.102040 4914 scope.go:122] "RemoveContainer" containerID="f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767" Nov 01 07:07:17.113139 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.113110 4914 scope.go:122] "RemoveContainer" containerID="f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767" Nov 01 07:07:17.113520 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.113493 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767\": container with ID starting with f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767 not found: ID does not exist" containerID="f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767" Nov 01 07:07:17.113624 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.113521 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767"} err="failed to get container status \"f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767\": rpc error: code = NotFound desc = could not find container \"f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767\": container with ID starting with f7d5ed7b915904f8d4f86491b6a49f4e9ffe6b8653872ce9af6006b9faab0767 not found: ID does not exist" Nov 01 07:07:17.129213 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129188 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.129355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129233 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129260 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.129478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129361 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129406 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129468 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.129645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129513 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.129645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129545 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129617 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129811 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129647 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129811 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129679 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.129811 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129717 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.129811 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129764 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.130014 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129830 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.130014 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129864 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.130014 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.129924 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh\") pod \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\" (UID: \"7aa800fd-cf83-4ea9-83f9-5130cae57a71\") " Nov 01 07:07:17.130014 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130003 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.130223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130030 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs\") pod \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\" (UID: \"6f9baa9b-e8f2-40f1-9b3c-780763f26864\") " Nov 01 07:07:17.130578 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130558 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.130661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130583 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.130661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130598 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.130661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130611 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.130661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130626 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7nr4\" (UniqueName: \"kubernetes.io/projected/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-kube-api-access-f7nr4\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.130661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130641 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.130661 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130654 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.131017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130668 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.131017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.130684 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.131534 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.131441 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.131869 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.131877 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.132006 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.132189 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.132193 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.132265 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.132477 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.132668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.132535 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.133743 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.133034 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.134180 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.133863 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:17.135171 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.135144 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.135501 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.135474 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:17.135704 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.135679 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:17.135861 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.135837 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "kube-api-access-bznzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:17.136114 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.136084 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:17.136914 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.136889 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs" pod "6f9baa9b-e8f2-40f1-9b3c-780763f26864" (UID: "6f9baa9b-e8f2-40f1-9b3c-780763f26864"). InnerVolumeSpecName "kube-api-access-wlfxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:17.137379 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.137358 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token" pod "7aa800fd-cf83-4ea9-83f9-5130cae57a71" (UID: "7aa800fd-cf83-4ea9-83f9-5130cae57a71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:17.232037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.231946 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.231978 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.231994 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232006 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232018 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bznzh\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-kube-api-access-bznzh\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232030 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232044 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlfxs\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-kube-api-access-wlfxs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232058 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.232064 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.232089 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-6f862af0425e: secret "agc-cred-6f862af0425e" not found Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.232096 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-dcba5d3eb853: secret "agc-cred-dcba5d3eb853" not found Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.232103 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-ea026b0e6c64: secret "agc-cred-ea026b0e6c64" not found Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.232116 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b: [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-dcba5d3eb853" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232072 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:17.232166 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials podName:c00a77b6-a06b-42a7-b825-c6cb64b40551 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:25.232151437 +0000 UTC m=+3460.784940639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551") : [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-dcba5d3eb853" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232183 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232195 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232211 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232225 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9baa9b-e8f2-40f1-9b3c-780763f26864-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232237 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9baa9b-e8f2-40f1-9b3c-780763f26864-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232247 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232256 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232264 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.232478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.232294 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa800fd-cf83-4ea9-83f9-5130cae57a71-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:17.301213 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.301155 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt"] Nov 01 07:07:17.303809 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.303781 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-l5fdt"] Nov 01 07:07:17.409515 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.409482 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857"] Nov 01 07:07:17.410467 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:07:17.410441 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea3598f_608f_43d1_bf70_24ed2f289ba1.slice/crio-533060f4b1b1e0fed1cab33aee97cd3b4de2aa63c6f3cfb0552013e68eb5be80 WatchSource:0}: Error finding container 533060f4b1b1e0fed1cab33aee97cd3b4de2aa63c6f3cfb0552013e68eb5be80: Status 404 returned error can't find the container with id 533060f4b1b1e0fed1cab33aee97cd3b4de2aa63c6f3cfb0552013e68eb5be80 Nov 01 07:07:17.676854 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.676819 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:07:17.677015 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.676910 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 07:07:17.980979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:17.980947 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl"] Nov 01 07:07:18.093152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.093121 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" Nov 01 07:07:18.093754 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.093164 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22" event={"ID":"7aa800fd-cf83-4ea9-83f9-5130cae57a71","Type":"ContainerDied","Data":"3c036a636e35e430e9d4f3166e26465352d55f23713c3642e4fcc114429e17b4"} Nov 01 07:07:18.093754 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.093230 4914 scope.go:122] "RemoveContainer" containerID="97b66d4a700c2618351709a24d35ed7dff6937b2a4b5579455acbe46ff1b5622" Nov 01 07:07:18.096385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.096079 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" event={"ID":"9ea3598f-608f-43d1-bf70-24ed2f289ba1","Type":"ContainerStarted","Data":"8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb"} Nov 01 07:07:18.096385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.096134 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:18.096385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.096156 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" event={"ID":"9ea3598f-608f-43d1-bf70-24ed2f289ba1","Type":"ContainerStarted","Data":"533060f4b1b1e0fed1cab33aee97cd3b4de2aa63c6f3cfb0552013e68eb5be80"} Nov 01 07:07:18.098454 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.098431 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk" Nov 01 07:07:18.290358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.287053 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" podStartSLOduration=2.287029764 podStartE2EDuration="2.287029764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:07:18.282109381 +0000 UTC m=+3453.834898576" watchObservedRunningTime="2026-11-01 07:07:18.287029764 +0000 UTC m=+3453.839818959" Nov 01 07:07:18.413382 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.413343 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk"] Nov 01 07:07:18.413710 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.413692 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-zxrpk"] Nov 01 07:07:18.529698 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.529667 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22"] Nov 01 07:07:18.534816 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.534790 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c831a4c0-openshift-default-cbd6b4ddf-b6x22"] Nov 01 07:07:18.617745 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.617716 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" path="/var/lib/kubelet/pods/6f9baa9b-e8f2-40f1-9b3c-780763f26864/volumes" Nov 01 07:07:18.618188 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.618176 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" path="/var/lib/kubelet/pods/7aa800fd-cf83-4ea9-83f9-5130cae57a71/volumes" Nov 01 07:07:18.618647 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.618634 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" path="/var/lib/kubelet/pods/dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f/volumes" Nov 01 07:07:18.677046 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.677017 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:07:18.677205 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:18.677064 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 07:07:19.677071 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:19.677033 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:07:19.677496 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:19.677083 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 07:07:20.676832 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:20.676791 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 01 07:07:20.677002 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:20.676852 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 01 07:07:21.385531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.385491 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" podUID="6b36c584-8644-4788-a7b2-1eedef0a51b3" containerName="payload-processing" containerID="cri-o://8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514" gracePeriod=25 Nov 01 07:07:21.640837 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.640771 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:21.681184 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.681159 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:21.681537 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.681518 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:21.682263 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.682242 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:07:21.717742 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.717719 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume\") pod \"6b36c584-8644-4788-a7b2-1eedef0a51b3\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " Nov 01 07:07:21.717810 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.717781 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7\") pod \"6b36c584-8644-4788-a7b2-1eedef0a51b3\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " Nov 01 07:07:21.717867 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.717853 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials\") pod \"6b36c584-8644-4788-a7b2-1eedef0a51b3\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " Nov 01 07:07:21.717979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.717955 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume\") pod \"6b36c584-8644-4788-a7b2-1eedef0a51b3\" (UID: \"6b36c584-8644-4788-a7b2-1eedef0a51b3\") " Nov 01 07:07:21.718119 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.718097 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume" pod "6b36c584-8644-4788-a7b2-1eedef0a51b3" (UID: "6b36c584-8644-4788-a7b2-1eedef0a51b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:21.718361 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.718329 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume" pod "6b36c584-8644-4788-a7b2-1eedef0a51b3" (UID: "6b36c584-8644-4788-a7b2-1eedef0a51b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:21.718740 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.718715 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:21.718740 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.718748 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b36c584-8644-4788-a7b2-1eedef0a51b3-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:21.720110 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.720094 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7" pod "6b36c584-8644-4788-a7b2-1eedef0a51b3" (UID: "6b36c584-8644-4788-a7b2-1eedef0a51b3"). InnerVolumeSpecName "kube-api-access-wk8v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:21.720225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.720206 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials" pod "6b36c584-8644-4788-a7b2-1eedef0a51b3" (UID: "6b36c584-8644-4788-a7b2-1eedef0a51b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:21.819576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.819538 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:21.819576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:21.819571 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk8v7\" (UniqueName: \"kubernetes.io/projected/6b36c584-8644-4788-a7b2-1eedef0a51b3-kube-api-access-wk8v7\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:22.128026 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.127996 4914 generic.go:356] "Generic (PLEG): container finished" podID="6b36c584-8644-4788-a7b2-1eedef0a51b3" containerID="8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514" exitCode=0 Nov 01 07:07:22.128175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.128038 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" Nov 01 07:07:22.128175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.128087 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" event={"ID":"6b36c584-8644-4788-a7b2-1eedef0a51b3","Type":"ContainerDied","Data":"8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514"} Nov 01 07:07:22.128175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.128134 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw" event={"ID":"6b36c584-8644-4788-a7b2-1eedef0a51b3","Type":"ContainerDied","Data":"f55f25939acce9e13b0d1e198454b26e875c6c20468aeb71ba8ee11158396949"} Nov 01 07:07:22.128175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.128162 4914 scope.go:122] "RemoveContainer" containerID="8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514" Nov 01 07:07:22.137949 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.137932 4914 scope.go:122] "RemoveContainer" containerID="8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514" Nov 01 07:07:22.138180 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:22.138161 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514\": container with ID starting with 8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514 not found: ID does not exist" containerID="8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514" Nov 01 07:07:22.138252 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.138186 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514"} err="failed to get container status \"8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514\": rpc error: code = NotFound desc = could not find container \"8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514\": container with ID starting with 8738cc12f67175748b19149429093012a55799e957864596f7b4bd057053d514 not found: ID does not exist" Nov 01 07:07:22.290783 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.290749 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw"] Nov 01 07:07:22.296353 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.296333 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-6cd769bzw"] Nov 01 07:07:22.617704 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.617677 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b36c584-8644-4788-a7b2-1eedef0a51b3" path="/var/lib/kubelet/pods/6b36c584-8644-4788-a7b2-1eedef0a51b3/volumes" Nov 01 07:07:22.981588 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:22.981504 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerName="payload-pre-processing" containerID="cri-o://f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b" gracePeriod=25 Nov 01 07:07:24.183976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:24.183936 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:25.255858 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:25.255825 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-522cd1e62a72: secret "agc-cred-522cd1e62a72" not found Nov 01 07:07:25.255858 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:25.255852 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-6f862af0425e: secret "agc-cred-6f862af0425e" not found Nov 01 07:07:25.255858 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:25.255859 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-dcba5d3eb853: secret "agc-cred-dcba5d3eb853" not found Nov 01 07:07:25.255858 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:25.255866 4914 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-caf74d/agc-cred-ea026b0e6c64: secret "agc-cred-ea026b0e6c64" not found Nov 01 07:07:25.256371 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:25.255879 4914 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b: [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-dcba5d3eb853" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:25.256371 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:25.255933 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials podName:c00a77b6-a06b-42a7-b825-c6cb64b40551 nodeName:}" failed. No retries permitted until 2026-11-01 07:07:41.255916829 +0000 UTC m=+3476.808706001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551") : [secret "agc-cred-522cd1e62a72" not found, secret "agc-cred-6f862af0425e" not found, secret "agc-cred-dcba5d3eb853" not found, secret "agc-cred-ea026b0e6c64" not found] Nov 01 07:07:26.064655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:26.064580 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:26.064812 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:26.064580 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:28.364028 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:28.363999 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b"] Nov 01 07:07:28.413635 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:28.413606 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857"] Nov 01 07:07:28.416475 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:28.416448 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" podUID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:29.622850 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.621503 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 07:07:29.715779 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.715755 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 07:07:29.723941 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.723923 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd\" (UniqueName: \"kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd\") pod \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " Nov 01 07:07:29.724126 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.724109 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume\") pod \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\" (UID: \"19dc3dd9-c337-49f6-82ea-3950b2c9b280\") " Nov 01 07:07:29.724507 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.724486 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume" pod "19dc3dd9-c337-49f6-82ea-3950b2c9b280" (UID: "19dc3dd9-c337-49f6-82ea-3950b2c9b280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:29.724632 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.724618 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19dc3dd9-c337-49f6-82ea-3950b2c9b280-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:29.726139 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.726118 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd" pod "19dc3dd9-c337-49f6-82ea-3950b2c9b280" (UID: "19dc3dd9-c337-49f6-82ea-3950b2c9b280"). InnerVolumeSpecName "kube-api-access-cxsvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:29.825384 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.825323 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg\" (UniqueName: \"kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg\") pod \"68459023-3b72-4d96-840f-b7ef5ace2865\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " Nov 01 07:07:29.825498 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.825473 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume\") pod \"68459023-3b72-4d96-840f-b7ef5ace2865\" (UID: \"68459023-3b72-4d96-840f-b7ef5ace2865\") " Nov 01 07:07:29.825822 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.825803 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsvd\" (UniqueName: \"kubernetes.io/projected/19dc3dd9-c337-49f6-82ea-3950b2c9b280-kube-api-access-cxsvd\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:29.825822 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.825809 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume" pod "68459023-3b72-4d96-840f-b7ef5ace2865" (UID: "68459023-3b72-4d96-840f-b7ef5ace2865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:29.827469 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.827450 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg" pod "68459023-3b72-4d96-840f-b7ef5ace2865" (UID: "68459023-3b72-4d96-840f-b7ef5ace2865"). InnerVolumeSpecName "kube-api-access-z87sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:29.927055 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.927027 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87sg\" (UniqueName: \"kubernetes.io/projected/68459023-3b72-4d96-840f-b7ef5ace2865-kube-api-access-z87sg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:29.927055 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:29.927054 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68459023-3b72-4d96-840f-b7ef5ace2865-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:30.179654 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.179621 4914 generic.go:356] "Generic (PLEG): container finished" podID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerID="2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07" exitCode=0 Nov 01 07:07:30.179843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.179669 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" Nov 01 07:07:30.179843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.179705 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" event={"ID":"19dc3dd9-c337-49f6-82ea-3950b2c9b280","Type":"ContainerDied","Data":"2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07"} Nov 01 07:07:30.179843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.179756 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj" event={"ID":"19dc3dd9-c337-49f6-82ea-3950b2c9b280","Type":"ContainerDied","Data":"baac14f3dca0a2b691cdf3943dd55ceb09995c87a0a154a13f30ab7648329256"} Nov 01 07:07:30.179843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.179778 4914 scope.go:122] "RemoveContainer" containerID="2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07" Nov 01 07:07:30.181102 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.181080 4914 generic.go:356] "Generic (PLEG): container finished" podID="68459023-3b72-4d96-840f-b7ef5ace2865" containerID="8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d" exitCode=0 Nov 01 07:07:30.181403 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.181110 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" event={"ID":"68459023-3b72-4d96-840f-b7ef5ace2865","Type":"ContainerDied","Data":"8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d"} Nov 01 07:07:30.181403 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.181148 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" Nov 01 07:07:30.181403 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.181152 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb" event={"ID":"68459023-3b72-4d96-840f-b7ef5ace2865","Type":"ContainerDied","Data":"30e6a1d43fa48e7c76a621afc8e18d798ca6c166c3aa978092f9a942734f9028"} Nov 01 07:07:30.192618 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.192599 4914 scope.go:122] "RemoveContainer" containerID="2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07" Nov 01 07:07:30.192950 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:30.192932 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07\": container with ID starting with 2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07 not found: ID does not exist" containerID="2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07" Nov 01 07:07:30.193007 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.192958 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07"} err="failed to get container status \"2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07\": rpc error: code = NotFound desc = could not find container \"2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07\": container with ID starting with 2169dc6e5b03a5ce569e9639f8104874681790d5033f731b54c6919b88f4ae07 not found: ID does not exist" Nov 01 07:07:30.193007 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.192975 4914 scope.go:122] "RemoveContainer" containerID="8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d" Nov 01 07:07:30.202580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.202561 4914 scope.go:122] "RemoveContainer" containerID="8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d" Nov 01 07:07:30.202846 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:30.202825 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d\": container with ID starting with 8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d not found: ID does not exist" containerID="8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d" Nov 01 07:07:30.202923 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.202856 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d"} err="failed to get container status \"8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d\": rpc error: code = NotFound desc = could not find container \"8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d\": container with ID starting with 8cd2cf92d5eb8ff2d7488293e0aa1f6eed09727a2f9fbcebad7178fd35fa0f7d not found: ID does not exist" Nov 01 07:07:30.351388 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.348325 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj"] Nov 01 07:07:30.351388 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.351021 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-302ba0-6879bdc87c-qqkcj"] Nov 01 07:07:30.455784 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.455714 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb"] Nov 01 07:07:30.462138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.462112 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-302ba0-77cd8985c9-fs5wb"] Nov 01 07:07:30.623266 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.623235 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" path="/var/lib/kubelet/pods/19dc3dd9-c337-49f6-82ea-3950b2c9b280/volumes" Nov 01 07:07:30.623720 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:30.623705 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" path="/var/lib/kubelet/pods/68459023-3b72-4d96-840f-b7ef5ace2865/volumes" Nov 01 07:07:33.365017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.364983 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" podUID="c00a77b6-a06b-42a7-b825-c6cb64b40551" containerName="payload-processing" containerID="cri-o://255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f" gracePeriod=25 Nov 01 07:07:33.414402 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.414369 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" podUID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" containerName="payload-processing" containerID="cri-o://8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb" gracePeriod=25 Nov 01 07:07:33.671750 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.671725 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 07:07:33.697218 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.697196 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:33.714468 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714445 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz\") pod \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " Nov 01 07:07:33.714576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714504 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw\") pod \"c00a77b6-a06b-42a7-b825-c6cb64b40551\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " Nov 01 07:07:33.714576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714545 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume\") pod \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " Nov 01 07:07:33.714576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714574 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume\") pod \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " Nov 01 07:07:33.714743 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714662 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials\") pod \"c00a77b6-a06b-42a7-b825-c6cb64b40551\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " Nov 01 07:07:33.714743 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714683 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume\") pod \"c00a77b6-a06b-42a7-b825-c6cb64b40551\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " Nov 01 07:07:33.714743 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714738 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume\") pod \"c00a77b6-a06b-42a7-b825-c6cb64b40551\" (UID: \"c00a77b6-a06b-42a7-b825-c6cb64b40551\") " Nov 01 07:07:33.714917 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714769 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials\") pod \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\" (UID: \"9ea3598f-608f-43d1-bf70-24ed2f289ba1\") " Nov 01 07:07:33.714972 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.714933 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume" pod "9ea3598f-608f-43d1-bf70-24ed2f289ba1" (UID: "9ea3598f-608f-43d1-bf70-24ed2f289ba1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:33.715362 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.715067 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume" pod "9ea3598f-608f-43d1-bf70-24ed2f289ba1" (UID: "9ea3598f-608f-43d1-bf70-24ed2f289ba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:33.715362 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.715139 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume" pod "c00a77b6-a06b-42a7-b825-c6cb64b40551" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:33.715362 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.715316 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume" pod "c00a77b6-a06b-42a7-b825-c6cb64b40551" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:33.715643 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.715384 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.715643 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.715401 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.715643 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.715412 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea3598f-608f-43d1-bf70-24ed2f289ba1-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.716891 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.716862 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw" pod "c00a77b6-a06b-42a7-b825-c6cb64b40551" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551"). InnerVolumeSpecName "kube-api-access-b8pmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:33.717081 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.717063 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials" pod "9ea3598f-608f-43d1-bf70-24ed2f289ba1" (UID: "9ea3598f-608f-43d1-bf70-24ed2f289ba1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:33.717373 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.717352 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz" pod "9ea3598f-608f-43d1-bf70-24ed2f289ba1" (UID: "9ea3598f-608f-43d1-bf70-24ed2f289ba1"). InnerVolumeSpecName "kube-api-access-wtxlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:33.717447 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.717384 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials" pod "c00a77b6-a06b-42a7-b825-c6cb64b40551" (UID: "c00a77b6-a06b-42a7-b825-c6cb64b40551"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:33.816576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.816547 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.816576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.816574 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00a77b6-a06b-42a7-b825-c6cb64b40551-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.816825 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.816585 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.816825 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.816594 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxlz\" (UniqueName: \"kubernetes.io/projected/9ea3598f-608f-43d1-bf70-24ed2f289ba1-kube-api-access-wtxlz\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:33.816825 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:33.816602 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8pmw\" (UniqueName: \"kubernetes.io/projected/c00a77b6-a06b-42a7-b825-c6cb64b40551-kube-api-access-b8pmw\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:34.182898 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.182861 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:34.207988 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.207957 4914 generic.go:356] "Generic (PLEG): container finished" podID="c00a77b6-a06b-42a7-b825-c6cb64b40551" containerID="255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f" exitCode=0 Nov 01 07:07:34.208125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.207998 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" Nov 01 07:07:34.208125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.208037 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" event={"ID":"c00a77b6-a06b-42a7-b825-c6cb64b40551","Type":"ContainerDied","Data":"255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f"} Nov 01 07:07:34.208125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.208096 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b" event={"ID":"c00a77b6-a06b-42a7-b825-c6cb64b40551","Type":"ContainerDied","Data":"62da585e37cba30022885d4adb5b318b319b5d72d774afa274148f1f87fdce68"} Nov 01 07:07:34.208125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.208119 4914 scope.go:122] "RemoveContainer" containerID="255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f" Nov 01 07:07:34.209655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.209632 4914 generic.go:356] "Generic (PLEG): container finished" podID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" containerID="8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb" exitCode=0 Nov 01 07:07:34.209771 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.209675 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" Nov 01 07:07:34.209771 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.209680 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" event={"ID":"9ea3598f-608f-43d1-bf70-24ed2f289ba1","Type":"ContainerDied","Data":"8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb"} Nov 01 07:07:34.209771 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.209715 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857" event={"ID":"9ea3598f-608f-43d1-bf70-24ed2f289ba1","Type":"ContainerDied","Data":"533060f4b1b1e0fed1cab33aee97cd3b4de2aa63c6f3cfb0552013e68eb5be80"} Nov 01 07:07:34.218554 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.218537 4914 scope.go:122] "RemoveContainer" containerID="255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f" Nov 01 07:07:34.218808 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:34.218779 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f\": container with ID starting with 255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f not found: ID does not exist" containerID="255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f" Nov 01 07:07:34.218898 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.218812 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f"} err="failed to get container status \"255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f\": rpc error: code = NotFound desc = could not find container \"255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f\": container with ID starting with 255b89d59e6043b83ed5f6cd30bc0f47bcebd2cc75f88cf8eb9bf35cd9b8257f not found: ID does not exist" Nov 01 07:07:34.218898 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.218837 4914 scope.go:122] "RemoveContainer" containerID="8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb" Nov 01 07:07:34.228338 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.228322 4914 scope.go:122] "RemoveContainer" containerID="8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb" Nov 01 07:07:34.228599 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:34.228578 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb\": container with ID starting with 8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb not found: ID does not exist" containerID="8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb" Nov 01 07:07:34.228684 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.228602 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb"} err="failed to get container status \"8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb\": rpc error: code = NotFound desc = could not find container \"8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb\": container with ID starting with 8af0691d0c64a754028fabd2b13e39cf0fae2e154ba8ab72a78fb571b1a5c3cb not found: ID does not exist" Nov 01 07:07:34.370079 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.370057 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857"] Nov 01 07:07:34.375296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.375251 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-85f9s4857"] Nov 01 07:07:34.483608 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.483547 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b"] Nov 01 07:07:34.487347 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.487322 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-caf74d/payload-processing-external-model-e2e-extmod-a-caf74d-87cbndj5b"] Nov 01 07:07:34.616843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.616813 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" path="/var/lib/kubelet/pods/9ea3598f-608f-43d1-bf70-24ed2f289ba1/volumes" Nov 01 07:07:34.617309 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:34.617292 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00a77b6-a06b-42a7-b825-c6cb64b40551" path="/var/lib/kubelet/pods/c00a77b6-a06b-42a7-b825-c6cb64b40551/volumes" Nov 01 07:07:37.600218 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:37.600171 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:07:43.257769 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.257743 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 07:07:43.268782 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.268712 4914 generic.go:356] "Generic (PLEG): container finished" podID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerID="f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b" exitCode=0 Nov 01 07:07:43.268931 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.268784 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" Nov 01 07:07:43.268989 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.268949 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" event={"ID":"a489380c-db6b-4d39-aacc-ceaf37ab5c81","Type":"ContainerDied","Data":"f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b"} Nov 01 07:07:43.269038 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.269006 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl" event={"ID":"a489380c-db6b-4d39-aacc-ceaf37ab5c81","Type":"ContainerDied","Data":"250bd354b2e986e550f24db1d8f1c23018c292e3031067f41a3107841b60a133"} Nov 01 07:07:43.269078 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.269040 4914 scope.go:122] "RemoveContainer" containerID="f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b" Nov 01 07:07:43.280786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.280768 4914 scope.go:122] "RemoveContainer" containerID="f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b" Nov 01 07:07:43.281098 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:43.281076 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b\": container with ID starting with f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b not found: ID does not exist" containerID="f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b" Nov 01 07:07:43.281203 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.281106 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b"} err="failed to get container status \"f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b\": rpc error: code = NotFound desc = could not find container \"f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b\": container with ID starting with f503060beff2268032750134779db2466ed0bb022c027f8f5942fca253a86c0b not found: ID does not exist" Nov 01 07:07:43.320312 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.320288 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5\" (UniqueName: \"kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5\") pod \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " Nov 01 07:07:43.320446 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.320369 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume\") pod \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\" (UID: \"a489380c-db6b-4d39-aacc-ceaf37ab5c81\") " Nov 01 07:07:43.320734 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.320712 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume" pod "a489380c-db6b-4d39-aacc-ceaf37ab5c81" (UID: "a489380c-db6b-4d39-aacc-ceaf37ab5c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:43.322365 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.322345 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5" pod "a489380c-db6b-4d39-aacc-ceaf37ab5c81" (UID: "a489380c-db6b-4d39-aacc-ceaf37ab5c81"). InnerVolumeSpecName "kube-api-access-47xh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:43.421233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.421204 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xh5\" (UniqueName: \"kubernetes.io/projected/a489380c-db6b-4d39-aacc-ceaf37ab5c81-kube-api-access-47xh5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:43.421233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.421230 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a489380c-db6b-4d39-aacc-ceaf37ab5c81-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:43.691698 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.691655 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl"] Nov 01 07:07:43.697645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:43.697612 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1ff1dc-8574466fb7-w5rsl"] Nov 01 07:07:44.617117 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:44.617081 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" path="/var/lib/kubelet/pods/a489380c-db6b-4d39-aacc-ceaf37ab5c81/volumes" Nov 01 07:07:44.653561 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:44.653531 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c"] Nov 01 07:07:44.653786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:44.653762 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" podUID="e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" containerName="maas-api" containerID="cri-o://38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57" gracePeriod=30 Nov 01 07:07:44.930613 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:44.930583 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 07:07:45.044303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.042445 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs\") pod \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " Nov 01 07:07:45.044303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.042499 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs\" (UniqueName: \"kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs\") pod \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " Nov 01 07:07:45.044303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.042667 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\") pod \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " Nov 01 07:07:45.044303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.042752 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca\") pod \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\" (UID: \"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b\") " Nov 01 07:07:45.044303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.043655 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca" pod "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" (UID: "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:45.065587 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.064160 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls" pod "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" (UID: "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:07:45.066895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.066845 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs" pod "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" (UID: "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:07:45.067248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.067221 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs" pod "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" (UID: "e5f792e6-c90a-43b4-b9a0-a5e01439eb7b"). InnerVolumeSpecName "kube-api-access-z46xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:45.144394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.144366 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:45.144394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.144394 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:45.144540 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.144403 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:45.144540 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.144414 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46xs\" (UniqueName: \"kubernetes.io/projected/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b-kube-api-access-z46xs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:45.282908 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.282821 4914 generic.go:356] "Generic (PLEG): container finished" podID="e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" containerID="38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57" exitCode=0 Nov 01 07:07:45.282908 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.282875 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" Nov 01 07:07:45.283091 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.282906 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" event={"ID":"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b","Type":"ContainerDied","Data":"38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57"} Nov 01 07:07:45.283091 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.282961 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c" event={"ID":"e5f792e6-c90a-43b4-b9a0-a5e01439eb7b","Type":"ContainerDied","Data":"b8fd092a1f533baf6c87209c9c46e74491578ff48b0a8d7800b6c35961c90696"} Nov 01 07:07:45.283091 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.282982 4914 scope.go:122] "RemoveContainer" containerID="38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57" Nov 01 07:07:45.293666 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.293650 4914 scope.go:122] "RemoveContainer" containerID="38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57" Nov 01 07:07:45.293942 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:45.293925 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57\": container with ID starting with 38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57 not found: ID does not exist" containerID="38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57" Nov 01 07:07:45.293988 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.293950 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57"} err="failed to get container status \"38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57\": rpc error: code = NotFound desc = could not find container \"38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57\": container with ID starting with 38fa23099b97d1a59d43fd53e91bf801421d9ca9225da62273b6ed1447b0ed57 not found: ID does not exist" Nov 01 07:07:45.446496 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.446467 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c"] Nov 01 07:07:45.494455 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:45.494424 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-302ba0-657c97498c-fr74c"] Nov 01 07:07:46.616452 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:46.616411 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" path="/var/lib/kubelet/pods/e5f792e6-c90a-43b4-b9a0-a5e01439eb7b/volumes" Nov 01 07:07:50.670589 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:50.670540 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq"] Nov 01 07:07:50.778791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:50.778755 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9"] Nov 01 07:07:52.773892 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:52.773858 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96"] Nov 01 07:07:52.774368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:52.774049 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" containerID="cri-o://f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732" gracePeriod=30 Nov 01 07:07:55.671670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:55.671624 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerName="payload-processing" containerID="cri-o://41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e" gracePeriod=25 Nov 01 07:07:55.780030 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:55.779970 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerName="payload-pre-processing" containerID="cri-o://cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d" gracePeriod=25 Nov 01 07:07:57.082430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:57.082388 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:57.082972 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:57.082582 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:07:58.045048 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.044978 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 07:07:58.100750 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.100714 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.100810 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.100853 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.100940 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.100983 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101051 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101089 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101116 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101200 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101150 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs\") pod \"5849375a-a8cc-4408-a9ec-962db5ebe542\" (UID: \"5849375a-a8cc-4408-a9ec-962db5ebe542\") " Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101217 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101270 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101410 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101613 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101654 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101662 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101688 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5849375a-a8cc-4408-a9ec-962db5ebe542-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.101717 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.101703 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.103202 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.103175 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:07:58.103353 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.103333 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:58.103554 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.103527 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "kube-api-access-85b9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:07:58.103554 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.103548 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy" pod "5849375a-a8cc-4408-a9ec-962db5ebe542" (UID: "5849375a-a8cc-4408-a9ec-962db5ebe542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:07:58.202343 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.202311 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.202343 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.202341 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.202343 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.202350 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85b9c\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-kube-api-access-85b9c\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.202575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.202360 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.202575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.202370 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.202575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.202378 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5849375a-a8cc-4408-a9ec-962db5ebe542-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:07:58.388510 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.388476 4914 generic.go:356] "Generic (PLEG): container finished" podID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerID="f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732" exitCode=0 Nov 01 07:07:58.388677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.388528 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" Nov 01 07:07:58.388677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.388600 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" event={"ID":"5849375a-a8cc-4408-a9ec-962db5ebe542","Type":"ContainerDied","Data":"f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732"} Nov 01 07:07:58.388677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.388651 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96" event={"ID":"5849375a-a8cc-4408-a9ec-962db5ebe542","Type":"ContainerDied","Data":"c32f91b9d16093c6ca1e9cf171a8eed9bb43e8918b545d8c721fe62814bb80a0"} Nov 01 07:07:58.388677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.388672 4914 scope.go:122] "RemoveContainer" containerID="f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732" Nov 01 07:07:58.399226 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.399205 4914 scope.go:122] "RemoveContainer" containerID="f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732" Nov 01 07:07:58.399524 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:07:58.399504 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732\": container with ID starting with f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732 not found: ID does not exist" containerID="f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732" Nov 01 07:07:58.399599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.399532 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732"} err="failed to get container status \"f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732\": rpc error: code = NotFound desc = could not find container \"f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732\": container with ID starting with f9d85525233be30e75a5fe6b9a5f15687d6fe11dbe942b85f63f8d0199a1d732 not found: ID does not exist" Nov 01 07:07:58.558410 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.558372 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96"] Nov 01 07:07:58.607755 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.607723 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-302ba0-openshift-default-5fd7ccb9ff-lmc96"] Nov 01 07:07:58.620447 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:07:58.620420 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" path="/var/lib/kubelet/pods/5849375a-a8cc-4408-a9ec-962db5ebe542/volumes" Nov 01 07:08:00.280733 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:00.280698 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw"] Nov 01 07:08:01.676188 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676151 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44"] Nov 01 07:08:01.676725 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676710 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c00a77b6-a06b-42a7-b825-c6cb64b40551" containerName="payload-processing" Nov 01 07:08:01.676725 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676725 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00a77b6-a06b-42a7-b825-c6cb64b40551" containerName="payload-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676736 4914 cpu_manager.go:414] "containerMap: removing container" podUID="6b36c584-8644-4788-a7b2-1eedef0a51b3" containerName="payload-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676744 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b36c584-8644-4788-a7b2-1eedef0a51b3" containerName="payload-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676769 4914 cpu_manager.go:414] "containerMap: removing container" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerName="payload-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676776 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerName="payload-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676786 4914 cpu_manager.go:414] "containerMap: removing container" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676791 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676800 4914 cpu_manager.go:414] "containerMap: removing container" podUID="e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" containerName="maas-api" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676805 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" containerName="maas-api" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676820 4914 cpu_manager.go:414] "containerMap: removing container" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" containerName="payload-pre-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676826 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" containerName="payload-pre-processing" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676846 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676853 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" Nov 01 07:08:01.676859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676861 4914 cpu_manager.go:414] "containerMap: removing container" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676867 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676874 4914 cpu_manager.go:414] "containerMap: removing container" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676879 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676890 4914 cpu_manager.go:414] "containerMap: removing container" podUID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" containerName="payload-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676894 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" containerName="payload-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676902 4914 cpu_manager.go:414] "containerMap: removing container" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerName="payload-pre-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676907 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerName="payload-pre-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.676989 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dc3dd9-c337-49f6-82ea-3950b2c9b280" containerName="payload-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677003 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa800fd-cf83-4ea9-83f9-5130cae57a71" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677014 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f792e6-c90a-43b4-b9a0-a5e01439eb7b" containerName="maas-api" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677021 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="68459023-3b72-4d96-840f-b7ef5ace2865" containerName="payload-pre-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677028 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5849375a-a8cc-4408-a9ec-962db5ebe542" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677033 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9baa9b-e8f2-40f1-9b3c-780763f26864" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677038 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="dca370f6-ce4f-45f3-b8c8-bf2b8e065f1f" containerName="istio-proxy" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677045 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea3598f-608f-43d1-bf70-24ed2f289ba1" containerName="payload-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677051 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a489380c-db6b-4d39-aacc-ceaf37ab5c81" containerName="payload-pre-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677057 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00a77b6-a06b-42a7-b825-c6cb64b40551" containerName="payload-processing" Nov 01 07:08:01.677248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.677062 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b36c584-8644-4788-a7b2-1eedef0a51b3" containerName="payload-processing" Nov 01 07:08:01.681858 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.681837 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.734552 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734516 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sr4l\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.734552 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734557 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.734818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734583 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.734818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734717 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.734818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734752 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.734818 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734815 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.735045 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734927 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.735045 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.734963 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.735149 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.735077 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.795994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.795961 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44"] Nov 01 07:08:01.836225 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836188 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr4l\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836247 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836289 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836322 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836593 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836406 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836593 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836504 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836593 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836582 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836758 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836620 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836758 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836707 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836758 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836734 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.836934 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.836775 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.837026 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.837000 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.837116 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.837100 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.837235 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.837213 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.838773 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.838747 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.839168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.839149 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.892162 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.892126 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.892458 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.892434 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sr4l\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9hv44\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:01.994604 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:01.994520 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:02.418749 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:02.418721 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44"] Nov 01 07:08:02.420075 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:08:02.420052 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd2cbe8_495b_4c88_9e09_a439ae9fc60b.slice/crio-86f0187088e63fafeca12a5ce93fc293ec09b53e4232f683d10aecd45927ffa8 WatchSource:0}: Error finding container 86f0187088e63fafeca12a5ce93fc293ec09b53e4232f683d10aecd45927ffa8: Status 404 returned error can't find the container with id 86f0187088e63fafeca12a5ce93fc293ec09b53e4232f683d10aecd45927ffa8 Nov 01 07:08:02.422413 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:02.422371 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:08:02.422491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:02.422446 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:08:02.422491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:02.422477 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:08:03.418242 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:03.418204 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" event={"ID":"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b","Type":"ContainerStarted","Data":"8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d"} Nov 01 07:08:03.418670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:03.418296 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" event={"ID":"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b","Type":"ContainerStarted","Data":"86f0187088e63fafeca12a5ce93fc293ec09b53e4232f683d10aecd45927ffa8"} Nov 01 07:08:03.995633 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:03.995602 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:03.996859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:03.996836 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9hv44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 07:08:03.996925 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:03.996899 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 07:08:04.995713 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:04.995677 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9hv44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 07:08:04.996114 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:04.995742 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 07:08:05.281407 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:05.281300 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerName="payload-pre-processing" containerID="cri-o://8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e" gracePeriod=25 Nov 01 07:08:05.995184 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:05.995144 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9hv44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 07:08:05.995404 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:05.995211 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 07:08:06.994964 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:06.994925 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9hv44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 07:08:06.995394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:06.994979 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 07:08:07.070427 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.070376 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:07.082287 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.082243 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:07.082432 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.082247 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:07.206491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.206424 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podStartSLOduration=6.206409303 podStartE2EDuration="6.206409303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:03.633458366 +0000 UTC m=+3499.186247560" watchObservedRunningTime="2026-11-01 07:08:07.206409303 +0000 UTC m=+3502.759198547" Nov 01 07:08:07.207381 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.207363 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn"] Nov 01 07:08:07.207583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.207562 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" podUID="8799c852-3724-4bb3-8576-9fb31e9638af" containerName="maas-api" containerID="cri-o://f9a58810da998dc4e4bd021887e0a10d56f4daf8610e1ee3d1f256de3d850576" gracePeriod=30 Nov 01 07:08:07.487347 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.487305 4914 generic.go:356] "Generic (PLEG): container finished" podID="8799c852-3724-4bb3-8576-9fb31e9638af" containerID="f9a58810da998dc4e4bd021887e0a10d56f4daf8610e1ee3d1f256de3d850576" exitCode=0 Nov 01 07:08:07.487515 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.487376 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" event={"ID":"8799c852-3724-4bb3-8576-9fb31e9638af","Type":"ContainerDied","Data":"f9a58810da998dc4e4bd021887e0a10d56f4daf8610e1ee3d1f256de3d850576"} Nov 01 07:08:07.571846 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.571813 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 07:08:07.623063 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:07.622915 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:08:07.632296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.632240 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s\" (UniqueName: \"kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s\") pod \"8799c852-3724-4bb3-8576-9fb31e9638af\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " Nov 01 07:08:07.633024 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.632998 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs\") pod \"8799c852-3724-4bb3-8576-9fb31e9638af\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " Nov 01 07:08:07.633363 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.633338 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\") pod \"8799c852-3724-4bb3-8576-9fb31e9638af\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " Nov 01 07:08:07.633563 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.633545 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca\") pod \"8799c852-3724-4bb3-8576-9fb31e9638af\" (UID: \"8799c852-3724-4bb3-8576-9fb31e9638af\") " Nov 01 07:08:07.633921 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.633893 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca" pod "8799c852-3724-4bb3-8576-9fb31e9638af" (UID: "8799c852-3724-4bb3-8576-9fb31e9638af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:07.634240 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.634222 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8799c852-3724-4bb3-8576-9fb31e9638af-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:07.637579 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.637545 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls" pod "8799c852-3724-4bb3-8576-9fb31e9638af" (UID: "8799c852-3724-4bb3-8576-9fb31e9638af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:07.637579 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.637555 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs" pod "8799c852-3724-4bb3-8576-9fb31e9638af" (UID: "8799c852-3724-4bb3-8576-9fb31e9638af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:07.638212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.638188 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s" pod "8799c852-3724-4bb3-8576-9fb31e9638af" (UID: "8799c852-3724-4bb3-8576-9fb31e9638af"). InnerVolumeSpecName "kube-api-access-t959s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:07.735816 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.735784 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t959s\" (UniqueName: \"kubernetes.io/projected/8799c852-3724-4bb3-8576-9fb31e9638af-kube-api-access-t959s\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:07.736012 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.735860 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:07.736012 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.735980 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8799c852-3724-4bb3-8576-9fb31e9638af-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:07.995186 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.995095 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9hv44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 07:08:07.995186 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:07.995151 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 07:08:08.494125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.494093 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" Nov 01 07:08:08.494313 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.494128 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn" event={"ID":"8799c852-3724-4bb3-8576-9fb31e9638af","Type":"ContainerDied","Data":"2fe78ae621d8c85286abaf75c836a42dea601662caad157c3e2dbdb59c19b4af"} Nov 01 07:08:08.494313 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.494194 4914 scope.go:122] "RemoveContainer" containerID="f9a58810da998dc4e4bd021887e0a10d56f4daf8610e1ee3d1f256de3d850576" Nov 01 07:08:08.656687 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.656653 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn"] Nov 01 07:08:08.658747 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.658726 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1ff1dc-8cb44959f-xrfpn"] Nov 01 07:08:08.995602 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.995564 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9hv44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 01 07:08:08.995994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:08.995628 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 01 07:08:10.618046 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:10.618011 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8799c852-3724-4bb3-8576-9fb31e9638af" path="/var/lib/kubelet/pods/8799c852-3724-4bb3-8576-9fb31e9638af/volumes" Nov 01 07:08:10.714285 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:10.714237 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:10.714573 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:10.714543 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:10.715297 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:10.715258 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:08:15.952345 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:15.947050 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 07:08:16.031733 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.031700 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44\" (UniqueName: \"kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44\") pod \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " Nov 01 07:08:16.031934 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.031810 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume\") pod \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\" (UID: \"93b5ec26-260b-4fa2-a798-a6bdb60a69a5\") " Nov 01 07:08:16.032306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.032256 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume" pod "93b5ec26-260b-4fa2-a798-a6bdb60a69a5" (UID: "93b5ec26-260b-4fa2-a798-a6bdb60a69a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:16.042915 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.042872 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44" pod "93b5ec26-260b-4fa2-a798-a6bdb60a69a5" (UID: "93b5ec26-260b-4fa2-a798-a6bdb60a69a5"). InnerVolumeSpecName "kube-api-access-mtm44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:16.066479 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.066444 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 07:08:16.132680 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.132642 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume\") pod \"c4daf58c-f842-4196-bdc0-6b180aa5a781\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " Nov 01 07:08:16.132902 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.132783 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr\" (UniqueName: \"kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr\") pod \"c4daf58c-f842-4196-bdc0-6b180aa5a781\" (UID: \"c4daf58c-f842-4196-bdc0-6b180aa5a781\") " Nov 01 07:08:16.133033 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.133017 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtm44\" (UniqueName: \"kubernetes.io/projected/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-kube-api-access-mtm44\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:16.133105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.133035 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93b5ec26-260b-4fa2-a798-a6bdb60a69a5-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:16.133164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.133127 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume" pod "c4daf58c-f842-4196-bdc0-6b180aa5a781" (UID: "c4daf58c-f842-4196-bdc0-6b180aa5a781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:16.135144 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.135116 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr" pod "c4daf58c-f842-4196-bdc0-6b180aa5a781" (UID: "c4daf58c-f842-4196-bdc0-6b180aa5a781"). InnerVolumeSpecName "kube-api-access-v9bnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:16.233930 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.233847 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4daf58c-f842-4196-bdc0-6b180aa5a781-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:16.233930 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.233877 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bnr\" (UniqueName: \"kubernetes.io/projected/c4daf58c-f842-4196-bdc0-6b180aa5a781-kube-api-access-v9bnr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:16.543960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.543872 4914 generic.go:356] "Generic (PLEG): container finished" podID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerID="cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d" exitCode=0 Nov 01 07:08:16.543960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.543921 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" Nov 01 07:08:16.543960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.543945 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" event={"ID":"c4daf58c-f842-4196-bdc0-6b180aa5a781","Type":"ContainerDied","Data":"cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d"} Nov 01 07:08:16.544227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.544024 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9" event={"ID":"c4daf58c-f842-4196-bdc0-6b180aa5a781","Type":"ContainerDied","Data":"2a87c99676be78db80c8021d85d7a863bd79c2057e6f2214d86d85bad506cdc4"} Nov 01 07:08:16.544227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.544052 4914 scope.go:122] "RemoveContainer" containerID="cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d" Nov 01 07:08:16.545424 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.545396 4914 generic.go:356] "Generic (PLEG): container finished" podID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerID="41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e" exitCode=0 Nov 01 07:08:16.545533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.545443 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" Nov 01 07:08:16.545533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.545470 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" event={"ID":"93b5ec26-260b-4fa2-a798-a6bdb60a69a5","Type":"ContainerDied","Data":"41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e"} Nov 01 07:08:16.545533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.545510 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq" event={"ID":"93b5ec26-260b-4fa2-a798-a6bdb60a69a5","Type":"ContainerDied","Data":"979cdba9182f4d061af4d856e6d936c1f2adc362b103d2121d682693f385e6eb"} Nov 01 07:08:16.554324 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.554199 4914 scope.go:122] "RemoveContainer" containerID="cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d" Nov 01 07:08:16.554606 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:16.554577 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d\": container with ID starting with cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d not found: ID does not exist" containerID="cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d" Nov 01 07:08:16.554678 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.554622 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d"} err="failed to get container status \"cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d\": rpc error: code = NotFound desc = could not find container \"cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d\": container with ID starting with cc5847b4dd7b5572be8630933ca13900718aaa33a63ca2617074c66961f7db0d not found: ID does not exist" Nov 01 07:08:16.554678 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.554648 4914 scope.go:122] "RemoveContainer" containerID="41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e" Nov 01 07:08:16.563939 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.563916 4914 scope.go:122] "RemoveContainer" containerID="41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e" Nov 01 07:08:16.564264 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:16.564243 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e\": container with ID starting with 41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e not found: ID does not exist" containerID="41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e" Nov 01 07:08:16.564344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.564288 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e"} err="failed to get container status \"41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e\": rpc error: code = NotFound desc = could not find container \"41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e\": container with ID starting with 41017f1b9ae2d22ee9d3cdf8bbcea558bf55b073620a932e7dd13da1ddd57e1e not found: ID does not exist" Nov 01 07:08:16.706615 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.706578 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq"] Nov 01 07:08:16.710254 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.710221 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-caf74d-55b6d48fb9-4mrdq"] Nov 01 07:08:16.776099 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776058 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn"] Nov 01 07:08:16.776748 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776726 4914 cpu_manager.go:414] "containerMap: removing container" podUID="8799c852-3724-4bb3-8576-9fb31e9638af" containerName="maas-api" Nov 01 07:08:16.776748 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776748 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8799c852-3724-4bb3-8576-9fb31e9638af" containerName="maas-api" Nov 01 07:08:16.776933 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776768 4914 cpu_manager.go:414] "containerMap: removing container" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerName="payload-processing" Nov 01 07:08:16.776933 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776775 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerName="payload-processing" Nov 01 07:08:16.776933 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776820 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerName="payload-pre-processing" Nov 01 07:08:16.776933 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776828 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerName="payload-pre-processing" Nov 01 07:08:16.777128 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.776982 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" containerName="payload-pre-processing" Nov 01 07:08:16.777128 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.777018 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8799c852-3724-4bb3-8576-9fb31e9638af" containerName="maas-api" Nov 01 07:08:16.777128 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.777028 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" containerName="payload-processing" Nov 01 07:08:16.781787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.781765 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.823759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.823674 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9"] Nov 01 07:08:16.827229 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.827197 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-caf74d-74b5f5bc75-9swk9"] Nov 01 07:08:16.837584 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.837549 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn"] Nov 01 07:08:16.840530 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840509 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840593 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840543 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840593 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840564 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840691 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840661 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840749 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840733 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840799 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840788 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqjq\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840836 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840813 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840838 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.840964 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.840952 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.941894 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.941859 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqjq\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.941894 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.941902 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.941983 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942096 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942163 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942193 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942248 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942298 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942334 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.942376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.942374 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.943135 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.943080 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.943668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.943392 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.943668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.943606 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.943810 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.943717 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.945753 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.945719 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.946033 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.946013 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.999051 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.999010 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:16.999479 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:16.999218 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqjq\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-7x5qn\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:17.061780 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.061741 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:17.094989 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.094901 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:17.529313 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.529253 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn"] Nov 01 07:08:17.530931 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:08:17.530903 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d08809f_cdc5_4f43_b478_7106de8beedb.slice/crio-77ee161b2f0154bde1cdf2fc132f285d3fe4b37ec881c3ae0c00813e33cf6f27 WatchSource:0}: Error finding container 77ee161b2f0154bde1cdf2fc132f285d3fe4b37ec881c3ae0c00813e33cf6f27: Status 404 returned error can't find the container with id 77ee161b2f0154bde1cdf2fc132f285d3fe4b37ec881c3ae0c00813e33cf6f27 Nov 01 07:08:17.533485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.533452 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:08:17.533568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.533522 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:08:17.533568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.533553 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:08:17.556510 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:17.556480 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" event={"ID":"1d08809f-cdc5-4f43-b478-7106de8beedb","Type":"ContainerStarted","Data":"77ee161b2f0154bde1cdf2fc132f285d3fe4b37ec881c3ae0c00813e33cf6f27"} Nov 01 07:08:18.575903 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:18.575863 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" event={"ID":"1d08809f-cdc5-4f43-b478-7106de8beedb","Type":"ContainerStarted","Data":"919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65"} Nov 01 07:08:18.616563 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:18.616531 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b5ec26-260b-4fa2-a798-a6bdb60a69a5" path="/var/lib/kubelet/pods/93b5ec26-260b-4fa2-a798-a6bdb60a69a5/volumes" Nov 01 07:08:18.616955 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:18.616942 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4daf58c-f842-4196-bdc0-6b180aa5a781" path="/var/lib/kubelet/pods/c4daf58c-f842-4196-bdc0-6b180aa5a781/volumes" Nov 01 07:08:18.741363 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:18.741300 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podStartSLOduration=2.741253307 podStartE2EDuration="2.741253307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:08:18.739899849 +0000 UTC m=+3514.292689042" watchObservedRunningTime="2026-11-01 07:08:18.741253307 +0000 UTC m=+3514.294042502" Nov 01 07:08:19.039401 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:19.039364 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2"] Nov 01 07:08:19.095078 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:19.095038 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:19.096673 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:19.096649 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:19.096779 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:19.096699 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:20.095350 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:20.095312 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:20.095751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:20.095368 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:21.095830 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:21.095793 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:21.096248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:21.095860 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:22.095780 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:22.095748 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:22.096071 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:22.095815 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:23.095955 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:23.095919 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:23.096137 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:23.095982 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:24.040258 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.040212 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" podUID="1e5940d4-266f-484e-8f49-1edbb557108a" containerName="payload-processing" containerID="cri-o://03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19" gracePeriod=25 Nov 01 07:08:24.096168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.096134 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:24.096571 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.096199 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:24.114831 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:08:24.114799 4914 logging.go:55] [core] [Channel #12793 SubChannel #12794]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.127:50052", ServerName: "10.132.0.127:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.127:50052: connect: connection refused" Nov 01 07:08:24.303549 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.303490 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 07:08:24.442065 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.442032 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h\" (UniqueName: \"kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h\") pod \"1e5940d4-266f-484e-8f49-1edbb557108a\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " Nov 01 07:08:24.442295 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.442073 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume\") pod \"1e5940d4-266f-484e-8f49-1edbb557108a\" (UID: \"1e5940d4-266f-484e-8f49-1edbb557108a\") " Nov 01 07:08:24.442616 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.442590 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume" pod "1e5940d4-266f-484e-8f49-1edbb557108a" (UID: "1e5940d4-266f-484e-8f49-1edbb557108a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:24.444508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.444480 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h" pod "1e5940d4-266f-484e-8f49-1edbb557108a" (UID: "1e5940d4-266f-484e-8f49-1edbb557108a"). InnerVolumeSpecName "kube-api-access-56p6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:24.543237 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.543200 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56p6h\" (UniqueName: \"kubernetes.io/projected/1e5940d4-266f-484e-8f49-1edbb557108a-kube-api-access-56p6h\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:24.543237 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.543236 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5940d4-266f-484e-8f49-1edbb557108a-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:24.631723 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.631690 4914 generic.go:356] "Generic (PLEG): container finished" podID="1e5940d4-266f-484e-8f49-1edbb557108a" containerID="03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19" exitCode=0 Nov 01 07:08:24.631880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.631739 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" Nov 01 07:08:24.631880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.631763 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" event={"ID":"1e5940d4-266f-484e-8f49-1edbb557108a","Type":"ContainerDied","Data":"03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19"} Nov 01 07:08:24.631880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.631814 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" event={"ID":"1e5940d4-266f-484e-8f49-1edbb557108a","Type":"ContainerDied","Data":"6f3de7516535f18d8d68a5d6ac9b322c8fb228383f725ad049ae36cf6b4a5dde"} Nov 01 07:08:24.631880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.631836 4914 scope.go:122] "RemoveContainer" containerID="03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19" Nov 01 07:08:24.643203 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.643178 4914 scope.go:122] "RemoveContainer" containerID="03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19" Nov 01 07:08:24.643570 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:24.643548 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19\": container with ID starting with 03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19 not found: ID does not exist" containerID="03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19" Nov 01 07:08:24.643647 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.643579 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19"} err="failed to get container status \"03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19\": rpc error: code = NotFound desc = could not find container \"03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19\": container with ID starting with 03f15ccb9be12ebfdbc5910d7dd0369589d4406a45889d025eb85b865977ed19 not found: ID does not exist" Nov 01 07:08:24.796623 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.796587 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2"] Nov 01 07:08:24.799812 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:24.799784 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2"] Nov 01 07:08:25.095666 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.095637 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-7x5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 01 07:08:25.095836 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.095686 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 01 07:08:25.114542 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.114502 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-581487-f88795567-8kgw2" podUID="1e5940d4-266f-484e-8f49-1edbb557108a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.127:50052\" within 1s: context deadline exceeded" Nov 01 07:08:25.562484 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.562456 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 07:08:25.644152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.644055 4914 generic.go:356] "Generic (PLEG): container finished" podID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerID="8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e" exitCode=0 Nov 01 07:08:25.644152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.644113 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" Nov 01 07:08:25.644152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.644137 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" event={"ID":"50f1aff2-ac8c-4d8e-a230-3ca057da816e","Type":"ContainerDied","Data":"8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e"} Nov 01 07:08:25.644493 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.644200 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw" event={"ID":"50f1aff2-ac8c-4d8e-a230-3ca057da816e","Type":"ContainerDied","Data":"c96ff1abbe40cd297964e79489456b01b702b41c4473b27443135e71beb575e4"} Nov 01 07:08:25.644493 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.644233 4914 scope.go:122] "RemoveContainer" containerID="8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e" Nov 01 07:08:25.655880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.655854 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume\") pod \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " Nov 01 07:08:25.656014 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.655941 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs\" (UniqueName: \"kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs\") pod \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\" (UID: \"50f1aff2-ac8c-4d8e-a230-3ca057da816e\") " Nov 01 07:08:25.656441 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.656409 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume" pod "50f1aff2-ac8c-4d8e-a230-3ca057da816e" (UID: "50f1aff2-ac8c-4d8e-a230-3ca057da816e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:25.657596 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.657567 4914 scope.go:122] "RemoveContainer" containerID="8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e" Nov 01 07:08:25.658023 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:25.657987 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e\": container with ID starting with 8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e not found: ID does not exist" containerID="8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e" Nov 01 07:08:25.658169 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.658028 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e"} err="failed to get container status \"8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e\": rpc error: code = NotFound desc = could not find container \"8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e\": container with ID starting with 8f60218f7fe0b4f7809f0fb78c0099f7d7ba71594f7ff5173460f881ae39112e not found: ID does not exist" Nov 01 07:08:25.658498 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.658477 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs" pod "50f1aff2-ac8c-4d8e-a230-3ca057da816e" (UID: "50f1aff2-ac8c-4d8e-a230-3ca057da816e"). InnerVolumeSpecName "kube-api-access-lqjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:25.757315 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.757260 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f1aff2-ac8c-4d8e-a230-3ca057da816e-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:25.757315 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:25.757302 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjjs\" (UniqueName: \"kubernetes.io/projected/50f1aff2-ac8c-4d8e-a230-3ca057da816e-kube-api-access-lqjjs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:26.067378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.067345 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw"] Nov 01 07:08:26.099738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.099712 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:26.100080 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.100059 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:26.100824 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.100804 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:08:26.116804 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.116781 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6ba6e5-5758557f59-tp7lw"] Nov 01 07:08:26.617457 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.617427 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5940d4-266f-484e-8f49-1edbb557108a" path="/var/lib/kubelet/pods/1e5940d4-266f-484e-8f49-1edbb557108a/volumes" Nov 01 07:08:26.617983 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:26.617956 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" path="/var/lib/kubelet/pods/50f1aff2-ac8c-4d8e-a230-3ca057da816e/volumes" Nov 01 07:08:28.391657 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:28.391622 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd"] Nov 01 07:08:28.499269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:28.498351 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d"] Nov 01 07:08:33.392876 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:33.392835 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerName="payload-processing" containerID="cri-o://5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20" gracePeriod=25 Nov 01 07:08:33.498873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:33.498808 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerName="payload-pre-processing" containerID="cri-o://b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6" gracePeriod=25 Nov 01 07:08:34.288425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:34.288390 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:34.288425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:34.288407 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:37.596352 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:37.596314 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:08:38.902906 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:38.902875 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4"] Nov 01 07:08:38.903391 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:38.903059 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" podUID="c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" containerName="maas-api" containerID="cri-o://5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5" gracePeriod=30 Nov 01 07:08:39.194096 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.194061 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 07:08:39.323609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.322504 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\") pod \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " Nov 01 07:08:39.323609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.322617 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs\") pod \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " Nov 01 07:08:39.323609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.322672 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca\") pod \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " Nov 01 07:08:39.323609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.322799 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z\" (UniqueName: \"kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z\") pod \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\" (UID: \"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c\") " Nov 01 07:08:39.327804 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.325868 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca" pod "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" (UID: "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:39.344733 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.343932 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z" pod "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" (UID: "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c"). InnerVolumeSpecName "kube-api-access-ntz8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:39.345041 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.344979 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs" pod "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" (UID: "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:39.346095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.346068 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls" pod "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" (UID: "c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:39.427379 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.427208 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntz8z\" (UniqueName: \"kubernetes.io/projected/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-kube-api-access-ntz8z\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:39.427379 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.427255 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:39.427379 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.427269 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:39.427379 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.427302 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:39.722550 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.722460 4914 generic.go:356] "Generic (PLEG): container finished" podID="c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" containerID="5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5" exitCode=0 Nov 01 07:08:39.722550 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.722510 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" Nov 01 07:08:39.722550 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.722531 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" event={"ID":"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c","Type":"ContainerDied","Data":"5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5"} Nov 01 07:08:39.722832 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.722573 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4" event={"ID":"c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c","Type":"ContainerDied","Data":"bf7afd502bd850e09031db68b7f27367ee132a3b6b2c94de639b4a6c7e3705b7"} Nov 01 07:08:39.722832 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.722593 4914 scope.go:122] "RemoveContainer" containerID="5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5" Nov 01 07:08:39.732095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.732076 4914 scope.go:122] "RemoveContainer" containerID="5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5" Nov 01 07:08:39.732417 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:39.732396 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5\": container with ID starting with 5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5 not found: ID does not exist" containerID="5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5" Nov 01 07:08:39.732490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.732423 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5"} err="failed to get container status \"5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5\": rpc error: code = NotFound desc = could not find container \"5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5\": container with ID starting with 5d3407d7957d7e924148dce7a156b29b2ece988503a535404453b8f6e4f31fc5 not found: ID does not exist" Nov 01 07:08:39.887587 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.887548 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4"] Nov 01 07:08:39.892565 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:39.892536 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ba6e5-54dfb654d5-wgmn4"] Nov 01 07:08:40.616227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:40.616192 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" path="/var/lib/kubelet/pods/c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c/volumes" Nov 01 07:08:44.287998 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:44.287956 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:44.288425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:44.287993 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:08:44.712105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:44.712068 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk"] Nov 01 07:08:44.712312 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:44.712263 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" podUID="80f62e03-6c2a-4002-b6e5-a0dd06c88917" containerName="maas-api" containerID="cri-o://1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615" gracePeriod=30 Nov 01 07:08:45.009428 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.009403 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 07:08:45.099327 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.099293 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2\" (UniqueName: \"kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2\") pod \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " Nov 01 07:08:45.099696 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.099672 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs\") pod \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " Nov 01 07:08:45.099868 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.099852 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca\") pod \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " Nov 01 07:08:45.100037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.100022 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\") pod \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\" (UID: \"80f62e03-6c2a-4002-b6e5-a0dd06c88917\") " Nov 01 07:08:45.107931 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.107836 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca" pod "80f62e03-6c2a-4002-b6e5-a0dd06c88917" (UID: "80f62e03-6c2a-4002-b6e5-a0dd06c88917"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:45.126697 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.126656 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2" pod "80f62e03-6c2a-4002-b6e5-a0dd06c88917" (UID: "80f62e03-6c2a-4002-b6e5-a0dd06c88917"). InnerVolumeSpecName "kube-api-access-hz8t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:45.126839 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.126751 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs" pod "80f62e03-6c2a-4002-b6e5-a0dd06c88917" (UID: "80f62e03-6c2a-4002-b6e5-a0dd06c88917"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:45.136307 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.135759 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls" pod "80f62e03-6c2a-4002-b6e5-a0dd06c88917" (UID: "80f62e03-6c2a-4002-b6e5-a0dd06c88917"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:45.201077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.201041 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:45.201077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.201074 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80f62e03-6c2a-4002-b6e5-a0dd06c88917-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:45.201293 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.201087 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80f62e03-6c2a-4002-b6e5-a0dd06c88917-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:45.201293 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.201099 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8t2\" (UniqueName: \"kubernetes.io/projected/80f62e03-6c2a-4002-b6e5-a0dd06c88917-kube-api-access-hz8t2\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:45.759055 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.759011 4914 generic.go:356] "Generic (PLEG): container finished" podID="80f62e03-6c2a-4002-b6e5-a0dd06c88917" containerID="1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615" exitCode=0 Nov 01 07:08:45.759598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.759054 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" event={"ID":"80f62e03-6c2a-4002-b6e5-a0dd06c88917","Type":"ContainerDied","Data":"1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615"} Nov 01 07:08:45.759598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.759079 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" Nov 01 07:08:45.759598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.759162 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk" event={"ID":"80f62e03-6c2a-4002-b6e5-a0dd06c88917","Type":"ContainerDied","Data":"21997c93d2f6985f760bd951874e9e6f1c9aaab34e7dc977ac7a477940f1cfc9"} Nov 01 07:08:45.759598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.759197 4914 scope.go:122] "RemoveContainer" containerID="1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615" Nov 01 07:08:45.770185 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.770164 4914 scope.go:122] "RemoveContainer" containerID="1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615" Nov 01 07:08:45.770494 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:45.770469 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615\": container with ID starting with 1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615 not found: ID does not exist" containerID="1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615" Nov 01 07:08:45.770591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.770499 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615"} err="failed to get container status \"1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615\": rpc error: code = NotFound desc = could not find container \"1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615\": container with ID starting with 1a9a2d339045f281329b7250d0740f10431ad53817e408cddff3cf2bd8941615 not found: ID does not exist" Nov 01 07:08:45.864461 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.864434 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p"] Nov 01 07:08:45.864722 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.864694 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" podUID="8d2a08b9-4216-4aa9-bb72-3976810f685a" containerName="maas-api" containerID="cri-o://e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650" gracePeriod=30 Nov 01 07:08:45.926259 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.926225 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk"] Nov 01 07:08:45.930907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:45.930881 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8ff537-5b8c4f6f5f-fcrvk"] Nov 01 07:08:46.160017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.159994 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 07:08:46.314199 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.314118 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca\") pod \"8d2a08b9-4216-4aa9-bb72-3976810f685a\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " Nov 01 07:08:46.314378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.314231 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\") pod \"8d2a08b9-4216-4aa9-bb72-3976810f685a\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " Nov 01 07:08:46.314378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.314298 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f\" (UniqueName: \"kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f\") pod \"8d2a08b9-4216-4aa9-bb72-3976810f685a\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " Nov 01 07:08:46.314506 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.314423 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs\") pod \"8d2a08b9-4216-4aa9-bb72-3976810f685a\" (UID: \"8d2a08b9-4216-4aa9-bb72-3976810f685a\") " Nov 01 07:08:46.314611 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.314579 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca" pod "8d2a08b9-4216-4aa9-bb72-3976810f685a" (UID: "8d2a08b9-4216-4aa9-bb72-3976810f685a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:46.314846 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.314820 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a08b9-4216-4aa9-bb72-3976810f685a-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:46.316697 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.316669 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f" pod "8d2a08b9-4216-4aa9-bb72-3976810f685a" (UID: "8d2a08b9-4216-4aa9-bb72-3976810f685a"). InnerVolumeSpecName "kube-api-access-jkn7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:46.317033 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.317006 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs" pod "8d2a08b9-4216-4aa9-bb72-3976810f685a" (UID: "8d2a08b9-4216-4aa9-bb72-3976810f685a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:46.317033 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.317017 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls" pod "8d2a08b9-4216-4aa9-bb72-3976810f685a" (UID: "8d2a08b9-4216-4aa9-bb72-3976810f685a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:08:46.416105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.415469 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:46.416105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.415508 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2a08b9-4216-4aa9-bb72-3976810f685a-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:46.416105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.415524 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkn7f\" (UniqueName: \"kubernetes.io/projected/8d2a08b9-4216-4aa9-bb72-3976810f685a-kube-api-access-jkn7f\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:46.615912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.615880 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f62e03-6c2a-4002-b6e5-a0dd06c88917" path="/var/lib/kubelet/pods/80f62e03-6c2a-4002-b6e5-a0dd06c88917/volumes" Nov 01 07:08:46.785668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.785619 4914 generic.go:356] "Generic (PLEG): container finished" podID="8d2a08b9-4216-4aa9-bb72-3976810f685a" containerID="e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650" exitCode=0 Nov 01 07:08:46.786152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.785688 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" event={"ID":"8d2a08b9-4216-4aa9-bb72-3976810f685a","Type":"ContainerDied","Data":"e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650"} Nov 01 07:08:46.786152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.785737 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" event={"ID":"8d2a08b9-4216-4aa9-bb72-3976810f685a","Type":"ContainerDied","Data":"999731002e7a05098bd8a33a054785173f0b4c6321da3c57c568836b14056c72"} Nov 01 07:08:46.786152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.785771 4914 scope.go:122] "RemoveContainer" containerID="e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650" Nov 01 07:08:46.786152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.785896 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p" Nov 01 07:08:46.831469 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.831426 4914 scope.go:122] "RemoveContainer" containerID="e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650" Nov 01 07:08:46.831965 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:46.831878 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650\": container with ID starting with e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650 not found: ID does not exist" containerID="e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650" Nov 01 07:08:46.831965 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.831923 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650"} err="failed to get container status \"e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650\": rpc error: code = NotFound desc = could not find container \"e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650\": container with ID starting with e94308ca69c3d23cc04264bbf1e5b311e6ee124fecb1b07210c39b28c1ad1650 not found: ID does not exist" Nov 01 07:08:46.952350 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:46.952251 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p"] Nov 01 07:08:47.000706 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:47.000676 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-581487-6646f654cc-t4m7p"] Nov 01 07:08:48.616315 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:48.616287 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2a08b9-4216-4aa9-bb72-3976810f685a" path="/var/lib/kubelet/pods/8d2a08b9-4216-4aa9-bb72-3976810f685a/volumes" Nov 01 07:08:53.662374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.662301 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 07:08:53.711009 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.710973 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg\" (UniqueName: \"kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg\") pod \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " Nov 01 07:08:53.711164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.711104 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume\") pod \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\" (UID: \"6802946e-4e90-4b1a-80bb-a04b17f3b2ae\") " Nov 01 07:08:53.711613 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.711581 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume" pod "6802946e-4e90-4b1a-80bb-a04b17f3b2ae" (UID: "6802946e-4e90-4b1a-80bb-a04b17f3b2ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:53.713261 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.713230 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg" pod "6802946e-4e90-4b1a-80bb-a04b17f3b2ae" (UID: "6802946e-4e90-4b1a-80bb-a04b17f3b2ae"). InnerVolumeSpecName "kube-api-access-4f7vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:53.753394 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.753369 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 07:08:53.811857 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.811825 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l\" (UniqueName: \"kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l\") pod \"8e69806a-6277-4edc-85a7-1d0038b53e30\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " Nov 01 07:08:53.812043 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.811926 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume\") pod \"8e69806a-6277-4edc-85a7-1d0038b53e30\" (UID: \"8e69806a-6277-4edc-85a7-1d0038b53e30\") " Nov 01 07:08:53.812306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.812259 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7vg\" (UniqueName: \"kubernetes.io/projected/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-kube-api-access-4f7vg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:53.812306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.812298 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6802946e-4e90-4b1a-80bb-a04b17f3b2ae-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:53.812422 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.812372 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume" pod "8e69806a-6277-4edc-85a7-1d0038b53e30" (UID: "8e69806a-6277-4edc-85a7-1d0038b53e30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:08:53.813894 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.813875 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l" pod "8e69806a-6277-4edc-85a7-1d0038b53e30" (UID: "8e69806a-6277-4edc-85a7-1d0038b53e30"). InnerVolumeSpecName "kube-api-access-7dm2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:08:53.825768 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.825732 4914 generic.go:356] "Generic (PLEG): container finished" podID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerID="5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20" exitCode=0 Nov 01 07:08:53.825934 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.825792 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" Nov 01 07:08:53.825934 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.825822 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" event={"ID":"6802946e-4e90-4b1a-80bb-a04b17f3b2ae","Type":"ContainerDied","Data":"5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20"} Nov 01 07:08:53.825934 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.825871 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd" event={"ID":"6802946e-4e90-4b1a-80bb-a04b17f3b2ae","Type":"ContainerDied","Data":"b513ff12a123f54f10c315f345631cd1fbe771d44377ec0e6d8ab67aa5c07f91"} Nov 01 07:08:53.825934 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.825898 4914 scope.go:122] "RemoveContainer" containerID="5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20" Nov 01 07:08:53.831176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.831149 4914 generic.go:356] "Generic (PLEG): container finished" podID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerID="b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6" exitCode=0 Nov 01 07:08:53.831320 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.831196 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" event={"ID":"8e69806a-6277-4edc-85a7-1d0038b53e30","Type":"ContainerDied","Data":"b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6"} Nov 01 07:08:53.831320 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.831208 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" Nov 01 07:08:53.831320 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.831234 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d" event={"ID":"8e69806a-6277-4edc-85a7-1d0038b53e30","Type":"ContainerDied","Data":"ffa63efc8a6ae6712ab389682ac4326b7909e77f5f4e27ec008dceaa6f79ec44"} Nov 01 07:08:53.839741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.839724 4914 scope.go:122] "RemoveContainer" containerID="5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20" Nov 01 07:08:53.840009 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:53.839981 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20\": container with ID starting with 5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20 not found: ID does not exist" containerID="5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20" Nov 01 07:08:53.840081 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.840020 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20"} err="failed to get container status \"5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20\": rpc error: code = NotFound desc = could not find container \"5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20\": container with ID starting with 5c083a89d84749d3a03af07f8cc884d4f21b1d29eec28bc510119169a1d8bb20 not found: ID does not exist" Nov 01 07:08:53.840081 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.840044 4914 scope.go:122] "RemoveContainer" containerID="b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6" Nov 01 07:08:53.848518 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.848501 4914 scope.go:122] "RemoveContainer" containerID="b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6" Nov 01 07:08:53.848785 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:08:53.848766 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6\": container with ID starting with b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6 not found: ID does not exist" containerID="b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6" Nov 01 07:08:53.848845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.848792 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6"} err="failed to get container status \"b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6\": rpc error: code = NotFound desc = could not find container \"b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6\": container with ID starting with b64e353943d10cece29d69ac9dba36a9398d4e939a8eede8847302dd014268b6 not found: ID does not exist" Nov 01 07:08:53.913845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.913764 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm2l\" (UniqueName: \"kubernetes.io/projected/8e69806a-6277-4edc-85a7-1d0038b53e30-kube-api-access-7dm2l\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:53.913845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:53.913819 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e69806a-6277-4edc-85a7-1d0038b53e30-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:08:54.000757 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:54.000715 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d"] Nov 01 07:08:54.004729 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:54.004700 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8ff537-7d88cbcd67-r8j8d"] Nov 01 07:08:54.111867 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:54.111836 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd"] Nov 01 07:08:54.115794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:54.115768 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-8ff537-5d69f665cc-hd8bd"] Nov 01 07:08:54.616774 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:54.616738 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" path="/var/lib/kubelet/pods/6802946e-4e90-4b1a-80bb-a04b17f3b2ae/volumes" Nov 01 07:08:54.617343 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:08:54.617322 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" path="/var/lib/kubelet/pods/8e69806a-6277-4edc-85a7-1d0038b53e30/volumes" Nov 01 07:09:00.058904 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.058820 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr"] Nov 01 07:09:00.059377 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059360 4914 cpu_manager.go:414] "containerMap: removing container" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerName="payload-pre-processing" Nov 01 07:09:00.059430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059379 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerName="payload-pre-processing" Nov 01 07:09:00.059430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059395 4914 cpu_manager.go:414] "containerMap: removing container" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerName="payload-processing" Nov 01 07:09:00.059430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059400 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerName="payload-processing" Nov 01 07:09:00.059430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059410 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1e5940d4-266f-484e-8f49-1edbb557108a" containerName="payload-processing" Nov 01 07:09:00.059430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059415 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5940d4-266f-484e-8f49-1edbb557108a" containerName="payload-processing" Nov 01 07:09:00.059430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059427 4914 cpu_manager.go:414] "containerMap: removing container" podUID="80f62e03-6c2a-4002-b6e5-a0dd06c88917" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059432 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f62e03-6c2a-4002-b6e5-a0dd06c88917" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059442 4914 cpu_manager.go:414] "containerMap: removing container" podUID="8d2a08b9-4216-4aa9-bb72-3976810f685a" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059447 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2a08b9-4216-4aa9-bb72-3976810f685a" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059463 4914 cpu_manager.go:414] "containerMap: removing container" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerName="payload-pre-processing" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059467 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerName="payload-pre-processing" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059475 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059480 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059574 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f1aff2-ac8c-4d8e-a230-3ca057da816e" containerName="payload-pre-processing" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059584 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6802946e-4e90-4b1a-80bb-a04b17f3b2ae" containerName="payload-processing" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059590 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5940d4-266f-484e-8f49-1edbb557108a" containerName="payload-processing" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059596 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f62e03-6c2a-4002-b6e5-a0dd06c88917" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059603 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2a08b9-4216-4aa9-bb72-3976810f685a" containerName="maas-api" Nov 01 07:09:00.059610 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059611 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e69806a-6277-4edc-85a7-1d0038b53e30" containerName="payload-pre-processing" Nov 01 07:09:00.059989 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.059617 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a94aea-9db5-47b5-9ee3-d9ec801b1a7c" containerName="maas-api" Nov 01 07:09:00.064074 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.064055 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.165188 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.165150 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8ff537-openshift-default-dockercfg-sbjxb\"" Nov 01 07:09:00.168189 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.168159 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr"] Nov 01 07:09:00.182004 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.181978 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182132 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182030 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182132 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182102 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182144 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182162 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182310 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182287 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182389 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182344 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lb8\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182475 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.182539 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.182514 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283394 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283431 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283638 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283615 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283679 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283649 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283747 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283730 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283767 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lb8\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283840 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283918 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283845 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.283918 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283882 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.284029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283927 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.284029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.283943 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.284152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.284027 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.284311 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.284269 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.284533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.284507 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.286197 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.286175 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.286355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.286339 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.341892 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.341836 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.341970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.341941 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lb8\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8\") pod \"e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.377750 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.377719 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:00.801906 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.801874 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr"] Nov 01 07:09:00.803970 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:09:00.803942 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b0982c_c9ea_4f25_a97f_17d1b9ba4a8b.slice/crio-c968b8a19e6046530f6ab0494c7f65c4ffce2172eccc35a5ded312695cda4715 WatchSource:0}: Error finding container c968b8a19e6046530f6ab0494c7f65c4ffce2172eccc35a5ded312695cda4715: Status 404 returned error can't find the container with id c968b8a19e6046530f6ab0494c7f65c4ffce2172eccc35a5ded312695cda4715 Nov 01 07:09:00.806243 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.806209 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:09:00.806351 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.806323 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:09:00.806392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.806357 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:09:00.891358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.887905 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr"] Nov 01 07:09:00.896992 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.896956 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" event={"ID":"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b","Type":"ContainerStarted","Data":"c968b8a19e6046530f6ab0494c7f65c4ffce2172eccc35a5ded312695cda4715"} Nov 01 07:09:00.897572 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:00.897542 4914 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" secret="" err="secret \"e2e-worker-w2-8ff537-openshift-default-dockercfg-sbjxb\" not found" Nov 01 07:09:01.015202 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:01.015139 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" podStartSLOduration=1.015124277 podStartE2EDuration="1.015124277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:01.011527205 +0000 UTC m=+3556.564316400" watchObservedRunningTime="2026-11-01 07:09:01.015124277 +0000 UTC m=+3556.567913473" Nov 01 07:09:01.378365 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:01.378319 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:01.379838 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:01.379816 4914 patch_prober.go:29] interesting pod/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 01 07:09:01.379937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:01.379865 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" podUID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 01 07:09:01.905345 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:01.905304 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" event={"ID":"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b","Type":"ContainerStarted","Data":"0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1"} Nov 01 07:09:01.905562 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:01.905394 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" podUID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" containerName="istio-proxy" containerID="cri-o://0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1" gracePeriod=30 Nov 01 07:09:07.184494 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.184467 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:07.280402 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280363 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280428 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280460 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280501 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280526 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280548 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280632 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280993 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280690 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280993 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280719 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data\") pod \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\" (UID: \"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b\") " Nov 01 07:09:07.280993 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280870 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:07.280993 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.280906 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:07.281202 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281132 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:07.281202 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281132 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:07.281202 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281144 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:07.281396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281264 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.281396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281321 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.281396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281332 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.281396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281343 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.281396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.281351 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.283058 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.283036 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:07.283231 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.283216 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:07.283471 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.283457 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:07.283515 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.283497 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8" pod "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" (UID: "59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b"). InnerVolumeSpecName "kube-api-access-57lb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:07.382136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.382095 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.382136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.382139 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.382368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.382156 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lb8\" (UniqueName: \"kubernetes.io/projected/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-kube-api-access-57lb8\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.382368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.382172 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:07.593455 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:07.593357 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:09:07.948405 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.948364 4914 generic.go:356] "Generic (PLEG): container finished" podID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" containerID="0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1" exitCode=0 Nov 01 07:09:07.948616 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.948416 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" Nov 01 07:09:07.948616 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.948447 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" event={"ID":"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b","Type":"ContainerDied","Data":"0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1"} Nov 01 07:09:07.948616 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.948514 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr" event={"ID":"59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b","Type":"ContainerDied","Data":"c968b8a19e6046530f6ab0494c7f65c4ffce2172eccc35a5ded312695cda4715"} Nov 01 07:09:07.948616 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.948543 4914 scope.go:122] "RemoveContainer" containerID="0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1" Nov 01 07:09:07.958838 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.958817 4914 scope.go:122] "RemoveContainer" containerID="0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1" Nov 01 07:09:07.959143 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:07.959118 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1\": container with ID starting with 0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1 not found: ID does not exist" containerID="0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1" Nov 01 07:09:07.959239 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:07.959148 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1"} err="failed to get container status \"0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1\": rpc error: code = NotFound desc = could not find container \"0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1\": container with ID starting with 0d73b4d7b386c42dcf95c672101c925954fe27067cc44c2dbba26ed8a1039bd1 not found: ID does not exist" Nov 01 07:09:08.111964 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:08.111931 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr"] Nov 01 07:09:08.161190 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:08.161154 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8ff537-openshift-default-69fb5bbbc9-q7hkr"] Nov 01 07:09:08.616508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:08.616472 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" path="/var/lib/kubelet/pods/59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b/volumes" Nov 01 07:09:16.666046 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.666008 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9"] Nov 01 07:09:16.666599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.666312 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" podUID="be12909e-3dac-4a7b-9fdd-dae945a16a44" containerName="maas-api" containerID="cri-o://e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509" gracePeriod=30 Nov 01 07:09:16.966201 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.966173 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 07:09:16.998608 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.998578 4914 generic.go:356] "Generic (PLEG): container finished" podID="be12909e-3dac-4a7b-9fdd-dae945a16a44" containerID="e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509" exitCode=0 Nov 01 07:09:16.998797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.998631 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" Nov 01 07:09:16.998797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.998676 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" event={"ID":"be12909e-3dac-4a7b-9fdd-dae945a16a44","Type":"ContainerDied","Data":"e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509"} Nov 01 07:09:16.998797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.998730 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9" event={"ID":"be12909e-3dac-4a7b-9fdd-dae945a16a44","Type":"ContainerDied","Data":"5e1f700daef1031dab29cac386aa2088d62fc0a1e1711c8095410d7ccb736ccc"} Nov 01 07:09:16.998797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:16.998754 4914 scope.go:122] "RemoveContainer" containerID="e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509" Nov 01 07:09:17.008214 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.008193 4914 scope.go:122] "RemoveContainer" containerID="e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509" Nov 01 07:09:17.008542 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:17.008522 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509\": container with ID starting with e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509 not found: ID does not exist" containerID="e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509" Nov 01 07:09:17.008628 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.008558 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509"} err="failed to get container status \"e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509\": rpc error: code = NotFound desc = could not find container \"e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509\": container with ID starting with e32fe7e2761bc99a6dacffd696d87b9372866a8d2938ad7e92e50ad7f9902509 not found: ID does not exist" Nov 01 07:09:17.121583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.121549 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs\") pod \"be12909e-3dac-4a7b-9fdd-dae945a16a44\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " Nov 01 07:09:17.121777 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.121600 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\") pod \"be12909e-3dac-4a7b-9fdd-dae945a16a44\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " Nov 01 07:09:17.121777 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.121628 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb\" (UniqueName: \"kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb\") pod \"be12909e-3dac-4a7b-9fdd-dae945a16a44\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " Nov 01 07:09:17.121777 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.121694 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca\") pod \"be12909e-3dac-4a7b-9fdd-dae945a16a44\" (UID: \"be12909e-3dac-4a7b-9fdd-dae945a16a44\") " Nov 01 07:09:17.122339 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.122255 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca" pod "be12909e-3dac-4a7b-9fdd-dae945a16a44" (UID: "be12909e-3dac-4a7b-9fdd-dae945a16a44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:17.124005 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.123974 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls" pod "be12909e-3dac-4a7b-9fdd-dae945a16a44" (UID: "be12909e-3dac-4a7b-9fdd-dae945a16a44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:17.124499 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.124481 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs" pod "be12909e-3dac-4a7b-9fdd-dae945a16a44" (UID: "be12909e-3dac-4a7b-9fdd-dae945a16a44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:17.124577 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.124540 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb" pod "be12909e-3dac-4a7b-9fdd-dae945a16a44" (UID: "be12909e-3dac-4a7b-9fdd-dae945a16a44"). InnerVolumeSpecName "kube-api-access-xq2tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:17.229410 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.228252 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be12909e-3dac-4a7b-9fdd-dae945a16a44-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:17.229410 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.228313 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:17.229410 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.228329 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be12909e-3dac-4a7b-9fdd-dae945a16a44-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:17.229410 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.228374 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq2tb\" (UniqueName: \"kubernetes.io/projected/be12909e-3dac-4a7b-9fdd-dae945a16a44-kube-api-access-xq2tb\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:17.418465 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.418427 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9"] Nov 01 07:09:17.423756 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:17.423728 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-190724-58c8d77c65-cllh9"] Nov 01 07:09:18.617204 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:18.617174 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be12909e-3dac-4a7b-9fdd-dae945a16a44" path="/var/lib/kubelet/pods/be12909e-3dac-4a7b-9fdd-dae945a16a44/volumes" Nov 01 07:09:29.421106 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421070 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9"] Nov 01 07:09:29.421656 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421636 4914 cpu_manager.go:414] "containerMap: removing container" podUID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" containerName="istio-proxy" Nov 01 07:09:29.421721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421660 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" containerName="istio-proxy" Nov 01 07:09:29.421721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421680 4914 cpu_manager.go:414] "containerMap: removing container" podUID="be12909e-3dac-4a7b-9fdd-dae945a16a44" containerName="maas-api" Nov 01 07:09:29.421721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421686 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be12909e-3dac-4a7b-9fdd-dae945a16a44" containerName="maas-api" Nov 01 07:09:29.421862 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421833 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b0982c-c9ea-4f25-a97f-17d1b9ba4a8b" containerName="istio-proxy" Nov 01 07:09:29.421862 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.421854 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="be12909e-3dac-4a7b-9fdd-dae945a16a44" containerName="maas-api" Nov 01 07:09:29.425138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.425119 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463246 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463210 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463260 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vglk\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463419 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463352 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463498 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463422 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463498 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463447 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463509 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463541 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463640 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463622 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.463679 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.463652 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.476887 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.476857 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-190724-openshift-default-dockercfg-vr7pw\"" Nov 01 07:09:29.564829 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.564782 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.564846 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vglk\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.564892 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.564929 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.564972 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565029 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.565008 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.565064 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.565111 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.565130 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.565604 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.565580 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.566127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.566099 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.566326 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.566116 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.566427 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.566138 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.566427 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.566165 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.568176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.568149 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.568656 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.568635 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.621621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.621592 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.621828 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.621805 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vglk\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk\") pod \"e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:29.627647 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.627610 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9"] Nov 01 07:09:29.738806 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:29.738699 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:30.165483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:30.165460 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9"] Nov 01 07:09:30.166747 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:09:30.166719 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4e20a7_daa6_45c7_b925_3b02a53a3d0d.slice/crio-9186107e118d918e05e1ee55c6c788e244d3a39554b0e3aa8a44dbd0b7f1397a WatchSource:0}: Error finding container 9186107e118d918e05e1ee55c6c788e244d3a39554b0e3aa8a44dbd0b7f1397a: Status 404 returned error can't find the container with id 9186107e118d918e05e1ee55c6c788e244d3a39554b0e3aa8a44dbd0b7f1397a Nov 01 07:09:30.168957 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:30.168913 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:09:30.169041 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:30.168990 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:09:30.169079 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:30.169037 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:09:31.087561 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:31.087516 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" event={"ID":"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d","Type":"ContainerStarted","Data":"f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162"} Nov 01 07:09:31.088023 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:31.087575 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" event={"ID":"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d","Type":"ContainerStarted","Data":"9186107e118d918e05e1ee55c6c788e244d3a39554b0e3aa8a44dbd0b7f1397a"} Nov 01 07:09:31.273022 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:31.272951 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podStartSLOduration=2.272920629 podStartE2EDuration="2.272920629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:09:31.268571657 +0000 UTC m=+3586.821360853" watchObservedRunningTime="2026-11-01 07:09:31.272920629 +0000 UTC m=+3586.825709824" Nov 01 07:09:31.739387 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:31.739347 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:31.741258 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:31.741232 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:09:31.741338 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:31.741316 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 07:09:32.739652 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:32.739611 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:09:32.740031 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:32.739672 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 07:09:33.739545 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:33.739511 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:09:33.739731 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:33.739578 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 07:09:34.739620 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:34.739578 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:09:34.739807 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:34.739647 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 07:09:35.739104 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:35.739071 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:09:35.739307 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:35.739136 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 07:09:36.739566 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:36.739532 4914 patch_prober.go:29] interesting pod/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 01 07:09:36.739985 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:36.739596 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 01 07:09:36.918403 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:36.918364 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx"] Nov 01 07:09:36.918705 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:36.918613 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" podUID="19fb1a00-b72d-47cc-92be-78f39f2efeeb" containerName="maas-api" containerID="cri-o://3278cee798fbf0e50627b25b2f038a9a7cf3469c503a5c1b40515810d049146e" gracePeriod=30 Nov 01 07:09:37.124145 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.124106 4914 generic.go:356] "Generic (PLEG): container finished" podID="19fb1a00-b72d-47cc-92be-78f39f2efeeb" containerID="3278cee798fbf0e50627b25b2f038a9a7cf3469c503a5c1b40515810d049146e" exitCode=0 Nov 01 07:09:37.124380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.124158 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" event={"ID":"19fb1a00-b72d-47cc-92be-78f39f2efeeb","Type":"ContainerDied","Data":"3278cee798fbf0e50627b25b2f038a9a7cf3469c503a5c1b40515810d049146e"} Nov 01 07:09:37.196220 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.196182 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9"] Nov 01 07:09:37.196543 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.196420 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" containerID="cri-o://f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162" gracePeriod=30 Nov 01 07:09:37.243972 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.243949 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 07:09:37.289520 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.289432 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\") pod \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " Nov 01 07:09:37.289681 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.289522 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs\") pod \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " Nov 01 07:09:37.289738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.289680 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca\") pod \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " Nov 01 07:09:37.289738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.289730 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf\" (UniqueName: \"kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf\") pod \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\" (UID: \"19fb1a00-b72d-47cc-92be-78f39f2efeeb\") " Nov 01 07:09:37.290228 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.290041 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca" pod "19fb1a00-b72d-47cc-92be-78f39f2efeeb" (UID: "19fb1a00-b72d-47cc-92be-78f39f2efeeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:37.290547 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.290525 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19fb1a00-b72d-47cc-92be-78f39f2efeeb-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:37.291970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.291947 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls" pod "19fb1a00-b72d-47cc-92be-78f39f2efeeb" (UID: "19fb1a00-b72d-47cc-92be-78f39f2efeeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:37.292085 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.291970 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs" pod "19fb1a00-b72d-47cc-92be-78f39f2efeeb" (UID: "19fb1a00-b72d-47cc-92be-78f39f2efeeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:09:37.292712 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.292690 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf" pod "19fb1a00-b72d-47cc-92be-78f39f2efeeb" (UID: "19fb1a00-b72d-47cc-92be-78f39f2efeeb"). InnerVolumeSpecName "kube-api-access-9tfnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:37.392089 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.392049 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tfnf\" (UniqueName: \"kubernetes.io/projected/19fb1a00-b72d-47cc-92be-78f39f2efeeb-kube-api-access-9tfnf\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:37.392089 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.392080 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:37.392089 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:37.392089 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19fb1a00-b72d-47cc-92be-78f39f2efeeb-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:37.593942 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:37.593856 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:09:38.131694 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:38.131665 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" Nov 01 07:09:38.132136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:38.131698 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx" event={"ID":"19fb1a00-b72d-47cc-92be-78f39f2efeeb","Type":"ContainerDied","Data":"f47cdccb1e2be1e4bbd1dd9fbbdde20c9f557efb1d5f27a7aedc7b341d2be9d9"} Nov 01 07:09:38.132136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:38.131765 4914 scope.go:122] "RemoveContainer" containerID="3278cee798fbf0e50627b25b2f038a9a7cf3469c503a5c1b40515810d049146e" Nov 01 07:09:38.293659 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:38.293623 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx"] Nov 01 07:09:38.343500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:38.343455 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ba2f3-66b58cc66f-gf5qx"] Nov 01 07:09:38.616623 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:38.616583 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fb1a00-b72d-47cc-92be-78f39f2efeeb" path="/var/lib/kubelet/pods/19fb1a00-b72d-47cc-92be-78f39f2efeeb/volumes" Nov 01 07:09:42.461575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.461539 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:42.550480 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550383 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550480 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550440 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550480 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550474 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550551 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550627 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550660 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550691 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.550781 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550757 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.551092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550790 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket\") pod \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\" (UID: \"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d\") " Nov 01 07:09:42.551092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550808 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:42.551092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550922 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.551092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550931 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.551092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.550953 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.551367 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.551140 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.551458 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.551425 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.551580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.551464 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.551580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.551478 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.551580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.551492 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.551580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.551520 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.552996 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.552967 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:42.553124 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.552998 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:42.553124 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.553009 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "kube-api-access-2vglk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:42.553254 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.553124 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy" pod "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" (UID: "2d4e20a7-daa6-45c7-b925-3b02a53a3d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:42.653341 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.653301 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vglk\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-kube-api-access-2vglk\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.653341 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.653340 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.653558 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.653355 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:42.653558 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:42.653370 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:43.162986 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.162950 4914 generic.go:356] "Generic (PLEG): container finished" podID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerID="f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162" exitCode=0 Nov 01 07:09:43.163183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.162997 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" Nov 01 07:09:43.163183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.163031 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" event={"ID":"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d","Type":"ContainerDied","Data":"f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162"} Nov 01 07:09:43.163183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.163088 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9" event={"ID":"2d4e20a7-daa6-45c7-b925-3b02a53a3d0d","Type":"ContainerDied","Data":"9186107e118d918e05e1ee55c6c788e244d3a39554b0e3aa8a44dbd0b7f1397a"} Nov 01 07:09:43.163183 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.163110 4914 scope.go:122] "RemoveContainer" containerID="f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162" Nov 01 07:09:43.172139 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.172120 4914 scope.go:122] "RemoveContainer" containerID="f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162" Nov 01 07:09:43.172460 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:43.172441 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162\": container with ID starting with f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162 not found: ID does not exist" containerID="f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162" Nov 01 07:09:43.172533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.172470 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162"} err="failed to get container status \"f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162\": rpc error: code = NotFound desc = could not find container \"f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162\": container with ID starting with f67ceffcfea78f22bf4e7043e613f43aa18a0a9a5823824b376b635b3a088162 not found: ID does not exist" Nov 01 07:09:43.375791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.375751 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9"] Nov 01 07:09:43.425303 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:43.425203 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-190724-openshift-default-58fbc9d9bc-ljmk9"] Nov 01 07:09:44.618127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:44.618085 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" path="/var/lib/kubelet/pods/2d4e20a7-daa6-45c7-b925-3b02a53a3d0d/volumes" Nov 01 07:09:45.075145 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:45.075043 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:09:45.087001 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:45.086566 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:09:48.167861 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:48.167828 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9"] Nov 01 07:09:48.168435 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:48.168031 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" containerID="cri-o://8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4" gracePeriod=30 Nov 01 07:09:48.182239 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:48.182208 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8"] Nov 01 07:09:48.182445 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:48.182421 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" containerID="cri-o://25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1" gracePeriod=30 Nov 01 07:09:51.684072 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:51.684037 4914 patch_prober.go:29] interesting pod/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 07:09:51.684491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:51.684100 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:09:53.444983 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.444906 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:09:53.475088 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.475061 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 07:09:53.604092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604053 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604106 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604130 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604161 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604435 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604357 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604435 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604401 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604519 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604452 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604519 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604508 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604630 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604535 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604630 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604585 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604626 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604655 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.604741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604683 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604724 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604751 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.604970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604790 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604855 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.604970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604887 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:53.604970 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604902 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs\") pod \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\" (UID: \"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5\") " Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604977 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data\") pod \"5e899976-d941-49b6-8c45-fdb3998f9d1b\" (UID: \"5e899976-d941-49b6-8c45-fdb3998f9d1b\") " Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.604997 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.605170 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.605452 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.605582 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.605852 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606080 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606175 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606245 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606267 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606303 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606319 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606334 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606351 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606365 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606381 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.606794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.606388 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.607791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.607293 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:53.607791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.607408 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:09:53.607791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.607456 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "kube-api-access-7skl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.607791 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.607770 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.608540 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.608514 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.608667 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.608566 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.608999 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.608966 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5" pod "5e899976-d941-49b6-8c45-fdb3998f9d1b" (UID: "5e899976-d941-49b6-8c45-fdb3998f9d1b"). InnerVolumeSpecName "kube-api-access-h6rh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:09:53.608999 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.608984 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy" pod "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" (UID: "49ea528e-6627-4cc2-b4a2-73fa7f47c2f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707865 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707907 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7skl6\" (UniqueName: \"kubernetes.io/projected/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-kube-api-access-7skl6\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707921 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707936 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707949 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707961 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707973 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rh5\" (UniqueName: \"kubernetes.io/projected/5e899976-d941-49b6-8c45-fdb3998f9d1b-kube-api-access-h6rh5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.707984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707985 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e899976-d941-49b6-8c45-fdb3998f9d1b-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.708489 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.707999 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e899976-d941-49b6-8c45-fdb3998f9d1b-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:53.708489 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:53.708012 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:09:54.232693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.232656 4914 generic.go:356] "Generic (PLEG): container finished" podID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerID="8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4" exitCode=0 Nov 01 07:09:54.232871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.232691 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" event={"ID":"5e899976-d941-49b6-8c45-fdb3998f9d1b","Type":"ContainerDied","Data":"8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4"} Nov 01 07:09:54.232871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.232707 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" Nov 01 07:09:54.232871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.232747 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9" event={"ID":"5e899976-d941-49b6-8c45-fdb3998f9d1b","Type":"ContainerDied","Data":"dcaafaf21f8063c1e4b13fbbf6983bb4983fea7d7928a4f38e0ce644d525e98d"} Nov 01 07:09:54.232871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.232771 4914 scope.go:122] "RemoveContainer" containerID="8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4" Nov 01 07:09:54.234664 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.234641 4914 generic.go:356] "Generic (PLEG): container finished" podID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerID="25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1" exitCode=0 Nov 01 07:09:54.234770 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.234688 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" Nov 01 07:09:54.234770 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.234728 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" event={"ID":"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5","Type":"ContainerDied","Data":"25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1"} Nov 01 07:09:54.234848 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.234773 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8" event={"ID":"49ea528e-6627-4cc2-b4a2-73fa7f47c2f5","Type":"ContainerDied","Data":"3089e41603010bdd593f8ab54829bf150963b3b970ec6a5a0ebcba10df52899f"} Nov 01 07:09:54.243872 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.243841 4914 scope.go:122] "RemoveContainer" containerID="8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4" Nov 01 07:09:54.244188 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:54.244165 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4\": container with ID starting with 8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4 not found: ID does not exist" containerID="8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4" Nov 01 07:09:54.244293 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.244196 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4"} err="failed to get container status \"8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4\": rpc error: code = NotFound desc = could not find container \"8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4\": container with ID starting with 8aef92754f490d17d620d8b1543a4da3faad32f37d3f0b218de28893e42dafd4 not found: ID does not exist" Nov 01 07:09:54.244293 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.244213 4914 scope.go:122] "RemoveContainer" containerID="25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1" Nov 01 07:09:54.253824 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.253790 4914 scope.go:122] "RemoveContainer" containerID="6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00" Nov 01 07:09:54.263871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.263843 4914 scope.go:122] "RemoveContainer" containerID="25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1" Nov 01 07:09:54.264315 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:54.264256 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1\": container with ID starting with 25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1 not found: ID does not exist" containerID="25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1" Nov 01 07:09:54.264406 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.264344 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1"} err="failed to get container status \"25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1\": rpc error: code = NotFound desc = could not find container \"25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1\": container with ID starting with 25add2acd735423732624eb4368811d35ec78f68371d500f4105fb6acf5a08d1 not found: ID does not exist" Nov 01 07:09:54.264406 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.264369 4914 scope.go:122] "RemoveContainer" containerID="6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00" Nov 01 07:09:54.264746 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:09:54.264721 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00\": container with ID starting with 6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00 not found: ID does not exist" containerID="6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00" Nov 01 07:09:54.264866 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.264751 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00"} err="failed to get container status \"6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00\": rpc error: code = NotFound desc = could not find container \"6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00\": container with ID starting with 6971ed3a99fcf35b18390dbe5da4a1118a77bf6961e56a4069105d86a0e1da00 not found: ID does not exist" Nov 01 07:09:54.394738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.394701 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9"] Nov 01 07:09:54.444137 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.444062 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-5dwv9"] Nov 01 07:09:54.551964 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.551925 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8"] Nov 01 07:09:54.556730 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.556703 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ba2f3-openshift-default-6db4fbcdbc-7khh8"] Nov 01 07:09:54.615803 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.615768 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" path="/var/lib/kubelet/pods/49ea528e-6627-4cc2-b4a2-73fa7f47c2f5/volumes" Nov 01 07:09:54.616205 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:09:54.616191 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" path="/var/lib/kubelet/pods/5e899976-d941-49b6-8c45-fdb3998f9d1b/volumes" Nov 01 07:10:07.591723 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:10:07.591684 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:10:37.592779 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:10:37.592742 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:11:07.592764 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:11:07.592726 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:11:37.610910 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:11:37.610874 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:12:07.593179 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:12:07.593087 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:12:37.592668 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:12:37.592629 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:13:07.592619 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:13:07.592561 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:13:37.592004 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:13:37.591914 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:13:48.390567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:48.390525 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh"] Nov 01 07:13:48.391107 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:48.390777 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" containerID="cri-o://bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06" gracePeriod=30 Nov 01 07:13:53.645454 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.645428 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 07:13:53.786569 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786499 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786569 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786557 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786583 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786649 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786696 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786729 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786988 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786780 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786988 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786866 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.786988 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.786907 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo\") pod \"c95ad465-57bc-4905-861a-cefcd9a6b717\" (UID: \"c95ad465-57bc-4905-861a-cefcd9a6b717\") " Nov 01 07:13:53.787160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.787060 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:13:53.787417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.787372 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.787550 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.787476 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:13:53.787708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.787664 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:13:53.787708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.787697 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:13:53.787817 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.787722 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:13:53.788907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.788886 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "kube-api-access-dt9kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:13:53.789319 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.789255 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:13:53.789468 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.789444 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:13:53.789667 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.789641 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo" pod "c95ad465-57bc-4905-861a-cefcd9a6b717" (UID: "c95ad465-57bc-4905-861a-cefcd9a6b717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:13:53.888059 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888032 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888059 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888057 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt9kr\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-kube-api-access-dt9kr\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888059 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888069 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888077 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c95ad465-57bc-4905-861a-cefcd9a6b717-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888086 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888096 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888104 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c95ad465-57bc-4905-861a-cefcd9a6b717-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:53.888301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:53.888112 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c95ad465-57bc-4905-861a-cefcd9a6b717-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:13:54.425092 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.425055 4914 generic.go:356] "Generic (PLEG): container finished" podID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerID="bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06" exitCode=0 Nov 01 07:13:54.425270 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.425109 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" Nov 01 07:13:54.425270 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.425145 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" event={"ID":"c95ad465-57bc-4905-861a-cefcd9a6b717","Type":"ContainerDied","Data":"bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06"} Nov 01 07:13:54.425270 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.425199 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh" event={"ID":"c95ad465-57bc-4905-861a-cefcd9a6b717","Type":"ContainerDied","Data":"a86924c97a03939d6addc45b7c89ae1221cf790974be4159cf4ead8380d583c1"} Nov 01 07:13:54.425270 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.425224 4914 scope.go:122] "RemoveContainer" containerID="bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06" Nov 01 07:13:54.435147 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.435129 4914 scope.go:122] "RemoveContainer" containerID="bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06" Nov 01 07:13:54.435536 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:13:54.435516 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06\": container with ID starting with bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06 not found: ID does not exist" containerID="bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06" Nov 01 07:13:54.435627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.435543 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06"} err="failed to get container status \"bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06\": rpc error: code = NotFound desc = could not find container \"bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06\": container with ID starting with bb4eb70af77ab000dfecf0009e3c8b9282ce0312a2cc71d5de5b60d1991bda06 not found: ID does not exist" Nov 01 07:13:54.616730 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.616707 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh"] Nov 01 07:13:54.667487 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:54.667456 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-28c598-openshift-default-7f67789789-ql6gh"] Nov 01 07:13:55.722597 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:55.722563 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6"] Nov 01 07:13:55.826691 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:55.826662 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v"] Nov 01 07:13:56.616065 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:13:56.616028 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" path="/var/lib/kubelet/pods/c95ad465-57bc-4905-861a-cefcd9a6b717/volumes" Nov 01 07:14:00.696783 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:00.696745 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg"] Nov 01 07:14:00.697321 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:00.696960 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" podUID="5af8892c-4390-43c2-af8e-8a898aed5b54" containerName="maas-api" containerID="cri-o://1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c" gracePeriod=30 Nov 01 07:14:00.723527 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:00.723489 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" podUID="0c472150-761c-43e0-b003-b323c493d208" containerName="payload-processing" containerID="cri-o://6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3" gracePeriod=25 Nov 01 07:14:00.827514 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:00.827478 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerName="payload-pre-processing" containerID="cri-o://f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1" gracePeriod=25 Nov 01 07:14:00.968608 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:00.968583 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 07:14:01.049380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.049352 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs\") pod \"5af8892c-4390-43c2-af8e-8a898aed5b54\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " Nov 01 07:14:01.049499 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.049452 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8\" (UniqueName: \"kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8\") pod \"5af8892c-4390-43c2-af8e-8a898aed5b54\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " Nov 01 07:14:01.049568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.049505 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\") pod \"5af8892c-4390-43c2-af8e-8a898aed5b54\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " Nov 01 07:14:01.049568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.049556 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca\") pod \"5af8892c-4390-43c2-af8e-8a898aed5b54\" (UID: \"5af8892c-4390-43c2-af8e-8a898aed5b54\") " Nov 01 07:14:01.049962 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.049936 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca" pod "5af8892c-4390-43c2-af8e-8a898aed5b54" (UID: "5af8892c-4390-43c2-af8e-8a898aed5b54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:14:01.051760 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.051732 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8" pod "5af8892c-4390-43c2-af8e-8a898aed5b54" (UID: "5af8892c-4390-43c2-af8e-8a898aed5b54"). InnerVolumeSpecName "kube-api-access-5c4n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:14:01.051856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.051770 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs" pod "5af8892c-4390-43c2-af8e-8a898aed5b54" (UID: "5af8892c-4390-43c2-af8e-8a898aed5b54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:14:01.051856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.051787 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls" pod "5af8892c-4390-43c2-af8e-8a898aed5b54" (UID: "5af8892c-4390-43c2-af8e-8a898aed5b54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:14:01.150422 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.150386 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:01.150422 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.150420 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4n8\" (UniqueName: \"kubernetes.io/projected/5af8892c-4390-43c2-af8e-8a898aed5b54-kube-api-access-5c4n8\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:01.150624 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.150437 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5af8892c-4390-43c2-af8e-8a898aed5b54-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:01.150624 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.150453 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af8892c-4390-43c2-af8e-8a898aed5b54-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:01.465047 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.465007 4914 generic.go:356] "Generic (PLEG): container finished" podID="5af8892c-4390-43c2-af8e-8a898aed5b54" containerID="1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c" exitCode=0 Nov 01 07:14:01.465223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.465058 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" Nov 01 07:14:01.465223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.465095 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" event={"ID":"5af8892c-4390-43c2-af8e-8a898aed5b54","Type":"ContainerDied","Data":"1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c"} Nov 01 07:14:01.465223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.465174 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg" event={"ID":"5af8892c-4390-43c2-af8e-8a898aed5b54","Type":"ContainerDied","Data":"95442d25ab872506a08a879814185e44e1c56702873953e8bb874f64c2991b9c"} Nov 01 07:14:01.465223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.465197 4914 scope.go:122] "RemoveContainer" containerID="1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c" Nov 01 07:14:01.474845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.474829 4914 scope.go:122] "RemoveContainer" containerID="1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c" Nov 01 07:14:01.475057 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:01.475040 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c\": container with ID starting with 1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c not found: ID does not exist" containerID="1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c" Nov 01 07:14:01.475125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.475064 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c"} err="failed to get container status \"1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c\": rpc error: code = NotFound desc = could not find container \"1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c\": container with ID starting with 1dce2b3ece6044c5b7e246b4a26a9ee6c39cf24bfd2ce60872c9b09cf0accc8c not found: ID does not exist" Nov 01 07:14:01.661192 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.661149 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg"] Nov 01 07:14:01.662895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:01.662868 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e0116-6c94cd65bc-cmllg"] Nov 01 07:14:02.616437 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:02.616406 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af8892c-4390-43c2-af8e-8a898aed5b54" path="/var/lib/kubelet/pods/5af8892c-4390-43c2-af8e-8a898aed5b54/volumes" Nov 01 07:14:04.980172 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:04.980142 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:04.980565 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:04.980142 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" podUID="0c472150-761c-43e0-b003-b323c493d208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:07.591820 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:07.591780 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:14:14.980580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:14.980543 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:14.980580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:14.980547 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" podUID="0c472150-761c-43e0-b003-b323c493d208" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:15.717982 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:15.717945 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct"] Nov 01 07:14:15.822834 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:15.822802 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg"] Nov 01 07:14:20.719266 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:20.719224 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" containerName="payload-processing" containerID="cri-o://0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99" gracePeriod=25 Nov 01 07:14:20.823153 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:20.823121 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerName="payload-pre-processing" containerID="cri-o://1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de" gracePeriod=25 Nov 01 07:14:20.968878 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:20.968841 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 07:14:21.068036 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.068010 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 07:14:21.138678 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.138653 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x\" (UniqueName: \"kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x\") pod \"0c472150-761c-43e0-b003-b323c493d208\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " Nov 01 07:14:21.138833 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.138704 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume\") pod \"0c472150-761c-43e0-b003-b323c493d208\" (UID: \"0c472150-761c-43e0-b003-b323c493d208\") " Nov 01 07:14:21.139106 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.139085 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume" pod "0c472150-761c-43e0-b003-b323c493d208" (UID: "0c472150-761c-43e0-b003-b323c493d208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:14:21.140871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.140842 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x" pod "0c472150-761c-43e0-b003-b323c493d208" (UID: "0c472150-761c-43e0-b003-b323c493d208"). InnerVolumeSpecName "kube-api-access-24q6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:14:21.239514 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.239489 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume\") pod \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " Nov 01 07:14:21.239636 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.239528 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw\" (UniqueName: \"kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw\") pod \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\" (UID: \"7b37d6d6-e5b3-465d-82c0-9a340cacd010\") " Nov 01 07:14:21.239859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.239671 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24q6x\" (UniqueName: \"kubernetes.io/projected/0c472150-761c-43e0-b003-b323c493d208-kube-api-access-24q6x\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:21.239859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.239681 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c472150-761c-43e0-b003-b323c493d208-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:21.239932 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.239882 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume" pod "7b37d6d6-e5b3-465d-82c0-9a340cacd010" (UID: "7b37d6d6-e5b3-465d-82c0-9a340cacd010"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:14:21.241684 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.241665 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw" pod "7b37d6d6-e5b3-465d-82c0-9a340cacd010" (UID: "7b37d6d6-e5b3-465d-82c0-9a340cacd010"). InnerVolumeSpecName "kube-api-access-jmkjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:14:21.340126 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.340073 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmkjw\" (UniqueName: \"kubernetes.io/projected/7b37d6d6-e5b3-465d-82c0-9a340cacd010-kube-api-access-jmkjw\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:21.340126 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.340096 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b37d6d6-e5b3-465d-82c0-9a340cacd010-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:21.570331 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.570270 4914 generic.go:356] "Generic (PLEG): container finished" podID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerID="f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1" exitCode=0 Nov 01 07:14:21.570490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.570343 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" Nov 01 07:14:21.570490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.570368 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" event={"ID":"7b37d6d6-e5b3-465d-82c0-9a340cacd010","Type":"ContainerDied","Data":"f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1"} Nov 01 07:14:21.570490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.570429 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v" event={"ID":"7b37d6d6-e5b3-465d-82c0-9a340cacd010","Type":"ContainerDied","Data":"96cee406ad09632d90858881cde3dba8da3388f53e355700f1fce27d5887f55c"} Nov 01 07:14:21.570490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.570457 4914 scope.go:122] "RemoveContainer" containerID="f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1" Nov 01 07:14:21.571875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.571847 4914 generic.go:356] "Generic (PLEG): container finished" podID="0c472150-761c-43e0-b003-b323c493d208" containerID="6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3" exitCode=0 Nov 01 07:14:21.571990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.571879 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" event={"ID":"0c472150-761c-43e0-b003-b323c493d208","Type":"ContainerDied","Data":"6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3"} Nov 01 07:14:21.571990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.571894 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" Nov 01 07:14:21.571990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.571931 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6" event={"ID":"0c472150-761c-43e0-b003-b323c493d208","Type":"ContainerDied","Data":"08fa4d1c0c16143e743fc682e6f674553faac868b7a5e92be72d120075d61e6d"} Nov 01 07:14:21.580721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.580702 4914 scope.go:122] "RemoveContainer" containerID="f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1" Nov 01 07:14:21.580993 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:21.580973 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1\": container with ID starting with f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1 not found: ID does not exist" containerID="f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1" Nov 01 07:14:21.581087 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.581006 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1"} err="failed to get container status \"f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1\": rpc error: code = NotFound desc = could not find container \"f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1\": container with ID starting with f5f80f60c1a99550dd25f84ebbb8ade144fccc36ef52903a3b6f396daea7f1e1 not found: ID does not exist" Nov 01 07:14:21.581087 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.581030 4914 scope.go:122] "RemoveContainer" containerID="6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3" Nov 01 07:14:21.589810 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.589789 4914 scope.go:122] "RemoveContainer" containerID="6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3" Nov 01 07:14:21.590070 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:21.590051 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3\": container with ID starting with 6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3 not found: ID does not exist" containerID="6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3" Nov 01 07:14:21.590156 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.590077 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3"} err="failed to get container status \"6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3\": rpc error: code = NotFound desc = could not find container \"6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3\": container with ID starting with 6778af3936af84049afc428b193a2a9bf4807fbe3f585fc0cb3c2787f5aac8b3 not found: ID does not exist" Nov 01 07:14:21.626211 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.626187 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn"] Nov 01 07:14:21.626384 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.626367 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" podUID="edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" containerName="maas-api" containerID="cri-o://d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2" gracePeriod=30 Nov 01 07:14:21.735121 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.735087 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6"] Nov 01 07:14:21.738725 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.738697 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2e0116-698dfd9db5-s5px6"] Nov 01 07:14:21.851740 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.851691 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v"] Nov 01 07:14:21.859528 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.859498 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2e0116-579d8c6cc9-v6h6v"] Nov 01 07:14:21.909197 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:21.909173 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 07:14:22.051694 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.051663 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca\") pod \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " Nov 01 07:14:22.051845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.051782 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97\" (UniqueName: \"kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97\") pod \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " Nov 01 07:14:22.051845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.051809 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\") pod \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " Nov 01 07:14:22.051979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.051871 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs\") pod \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\" (UID: \"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf\") " Nov 01 07:14:22.052146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.052119 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca" pod "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" (UID: "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:14:22.054098 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.054071 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls" pod "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" (UID: "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:14:22.054264 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.054248 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97" pod "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" (UID: "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf"). InnerVolumeSpecName "kube-api-access-ppd97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:14:22.054396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.054372 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs" pod "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" (UID: "edcdd90e-cdf8-4c72-a634-a50cd15e9cbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:14:22.152942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.152870 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:22.152942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.152898 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppd97\" (UniqueName: \"kubernetes.io/projected/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-kube-api-access-ppd97\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:22.152942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.152914 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:22.152942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.152926 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:22.585830 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.585737 4914 generic.go:356] "Generic (PLEG): container finished" podID="edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" containerID="d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2" exitCode=0 Nov 01 07:14:22.586017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.585933 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" Nov 01 07:14:22.587174 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.587119 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" event={"ID":"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf","Type":"ContainerDied","Data":"d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2"} Nov 01 07:14:22.588066 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.587996 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn" event={"ID":"edcdd90e-cdf8-4c72-a634-a50cd15e9cbf","Type":"ContainerDied","Data":"c6d166ee4b79aa5dd512f2b34f54d359887313c116d7d99dee1b559157975dfa"} Nov 01 07:14:22.588066 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.588050 4914 scope.go:122] "RemoveContainer" containerID="d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2" Nov 01 07:14:22.602343 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.602317 4914 scope.go:122] "RemoveContainer" containerID="d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2" Nov 01 07:14:22.602751 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:22.602681 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2\": container with ID starting with d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2 not found: ID does not exist" containerID="d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2" Nov 01 07:14:22.602838 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.602765 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2"} err="failed to get container status \"d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2\": rpc error: code = NotFound desc = could not find container \"d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2\": container with ID starting with d5edbe332739b981c64f7e4c7fde795e8f21cf962244872af7d75215a17e91a2 not found: ID does not exist" Nov 01 07:14:22.618165 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.618136 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c472150-761c-43e0-b003-b323c493d208" path="/var/lib/kubelet/pods/0c472150-761c-43e0-b003-b323c493d208/volumes" Nov 01 07:14:22.618570 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.618555 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" path="/var/lib/kubelet/pods/7b37d6d6-e5b3-465d-82c0-9a340cacd010/volumes" Nov 01 07:14:22.748484 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.748452 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn"] Nov 01 07:14:22.797802 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:22.797778 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-51aa8f-6b88cc5bf-xd4pn"] Nov 01 07:14:24.616129 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:24.616091 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" path="/var/lib/kubelet/pods/edcdd90e-cdf8-4c72-a634-a50cd15e9cbf/volumes" Nov 01 07:14:27.777976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:27.777933 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:27.778437 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:27.777939 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:33.072907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:33.072878 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575f85f67b-6vdks_52b6aac2-2f3b-48e1-b1b1-f0445358abe8/manager/0.log" Nov 01 07:14:34.944243 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:34.944215 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8p824_12e2413a-0690-4528-8f90-94f6bd824f63/server/0.log" Nov 01 07:14:37.593259 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:37.593223 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:14:37.777945 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:37.777906 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:37.777945 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:37.777924 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:14:40.888789 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.888756 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl"] Nov 01 07:14:40.889290 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889257 4914 cpu_manager.go:414] "containerMap: removing container" podUID="19fb1a00-b72d-47cc-92be-78f39f2efeeb" containerName="maas-api" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889294 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fb1a00-b72d-47cc-92be-78f39f2efeeb" containerName="maas-api" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889320 4914 cpu_manager.go:414] "containerMap: removing container" podUID="0c472150-761c-43e0-b003-b323c493d208" containerName="payload-processing" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889329 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c472150-761c-43e0-b003-b323c493d208" containerName="payload-processing" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889339 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5af8892c-4390-43c2-af8e-8a898aed5b54" containerName="maas-api" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889344 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af8892c-4390-43c2-af8e-8a898aed5b54" containerName="maas-api" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889359 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889363 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" Nov 01 07:14:40.889372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889372 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889378 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889386 4914 cpu_manager.go:414] "containerMap: removing container" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerName="payload-pre-processing" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889392 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerName="payload-pre-processing" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889400 4914 cpu_manager.go:414] "containerMap: removing container" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889404 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889416 4914 cpu_manager.go:414] "containerMap: removing container" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889421 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889432 4914 cpu_manager.go:414] "containerMap: removing container" podUID="edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" containerName="maas-api" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889436 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" containerName="maas-api" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889442 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889446 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889540 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcdd90e-cdf8-4c72-a634-a50cd15e9cbf" containerName="maas-api" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889549 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e899976-d941-49b6-8c45-fdb3998f9d1b" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889557 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fb1a00-b72d-47cc-92be-78f39f2efeeb" containerName="maas-api" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889563 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c472150-761c-43e0-b003-b323c493d208" containerName="payload-processing" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889568 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af8892c-4390-43c2-af8e-8a898aed5b54" containerName="maas-api" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889574 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4e20a7-daa6-45c7-b925-3b02a53a3d0d" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889582 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95ad465-57bc-4905-861a-cefcd9a6b717" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889590 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b37d6d6-e5b3-465d-82c0-9a340cacd010" containerName="payload-pre-processing" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889600 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" Nov 01 07:14:40.889786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.889607 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ea528e-6627-4cc2-b4a2-73fa7f47c2f5" containerName="istio-proxy" Nov 01 07:14:40.893351 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.892925 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:40.949476 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.949438 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-976689-openshift-default-dockercfg-k9llf\"" Nov 01 07:14:40.986537 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.986473 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 07:14:40.998739 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:40.998502 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl"] Nov 01 07:14:41.014629 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.014597 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume\") pod \"b4f27112-de78-4352-bee3-b1533e827e0d\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " Nov 01 07:14:41.014769 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.014652 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5\" (UniqueName: \"kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5\") pod \"b4f27112-de78-4352-bee3-b1533e827e0d\" (UID: \"b4f27112-de78-4352-bee3-b1533e827e0d\") " Nov 01 07:14:41.014904 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.014885 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hxh\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.014950 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.014932 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015058 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015021 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume" pod "b4f27112-de78-4352-bee3-b1533e827e0d" (UID: "b4f27112-de78-4352-bee3-b1533e827e0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:14:41.015058 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015033 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015083 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015165 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015314 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015206 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015314 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015236 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015597 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015363 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015597 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015473 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.015597 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.015555 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f27112-de78-4352-bee3-b1533e827e0d-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:41.017124 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.017100 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5" pod "b4f27112-de78-4352-bee3-b1533e827e0d" (UID: "b4f27112-de78-4352-bee3-b1533e827e0d"). InnerVolumeSpecName "kube-api-access-6fxr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:14:41.101990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.101955 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 07:14:41.116385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116356 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hxh\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116498 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116396 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116549 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116525 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116600 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116579 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116654 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116602 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116654 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116627 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116669 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116713 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116783 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.116875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116870 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fxr5\" (UniqueName: \"kubernetes.io/projected/b4f27112-de78-4352-bee3-b1533e827e0d-kube-api-access-6fxr5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:41.117057 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.116962 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.117057 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.117019 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.117141 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.117071 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.117177 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.117129 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.117607 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.117561 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.119036 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.119016 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.119174 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.119156 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.172412 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.172385 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.172526 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.172486 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hxh\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh\") pod \"e2e-worker-main-976689-openshift-default-5db554f479-mgddl\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.215726 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.215702 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:41.217943 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.217925 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2\" (UniqueName: \"kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2\") pod \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " Nov 01 07:14:41.218024 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.218009 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume\") pod \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\" (UID: \"f90e61ce-cc69-4f71-b18a-23b8da40aeab\") " Nov 01 07:14:41.218350 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.218326 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume" pod "f90e61ce-cc69-4f71-b18a-23b8da40aeab" (UID: "f90e61ce-cc69-4f71-b18a-23b8da40aeab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:14:41.219940 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.219916 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2" pod "f90e61ce-cc69-4f71-b18a-23b8da40aeab" (UID: "f90e61ce-cc69-4f71-b18a-23b8da40aeab"). InnerVolumeSpecName "kube-api-access-qwcz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:14:41.319244 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.319214 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcz2\" (UniqueName: \"kubernetes.io/projected/f90e61ce-cc69-4f71-b18a-23b8da40aeab-kube-api-access-qwcz2\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:41.319401 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.319246 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90e61ce-cc69-4f71-b18a-23b8da40aeab-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:14:41.658252 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.658208 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl"] Nov 01 07:14:41.660709 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:14:41.660679 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2cc16e_c9b1_4e60_b632_c824124630c7.slice/crio-e9e1d217364def26b79069fab1fe1186344a0c1b3e64c4a35cca33b593054b2e WatchSource:0}: Error finding container e9e1d217364def26b79069fab1fe1186344a0c1b3e64c4a35cca33b593054b2e: Status 404 returned error can't find the container with id e9e1d217364def26b79069fab1fe1186344a0c1b3e64c4a35cca33b593054b2e Nov 01 07:14:41.662845 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.662808 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:14:41.662924 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.662879 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:14:41.662924 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.662906 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:14:41.699473 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.699448 4914 generic.go:356] "Generic (PLEG): container finished" podID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerID="1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de" exitCode=0 Nov 01 07:14:41.699583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.699493 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" Nov 01 07:14:41.699583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.699517 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" event={"ID":"f90e61ce-cc69-4f71-b18a-23b8da40aeab","Type":"ContainerDied","Data":"1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de"} Nov 01 07:14:41.699583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.699571 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg" event={"ID":"f90e61ce-cc69-4f71-b18a-23b8da40aeab","Type":"ContainerDied","Data":"8d6908c6bfa003d91bd11762fe15cb1ad3a7163d8822fbb3eef770030bd93ee3"} Nov 01 07:14:41.699833 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.699603 4914 scope.go:122] "RemoveContainer" containerID="1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de" Nov 01 07:14:41.700880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.700847 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" event={"ID":"1d2cc16e-c9b1-4e60-b632-c824124630c7","Type":"ContainerStarted","Data":"e9e1d217364def26b79069fab1fe1186344a0c1b3e64c4a35cca33b593054b2e"} Nov 01 07:14:41.702128 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.702105 4914 generic.go:356] "Generic (PLEG): container finished" podID="b4f27112-de78-4352-bee3-b1533e827e0d" containerID="0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99" exitCode=0 Nov 01 07:14:41.702221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.702148 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" Nov 01 07:14:41.702221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.702183 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" event={"ID":"b4f27112-de78-4352-bee3-b1533e827e0d","Type":"ContainerDied","Data":"0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99"} Nov 01 07:14:41.702221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.702217 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct" event={"ID":"b4f27112-de78-4352-bee3-b1533e827e0d","Type":"ContainerDied","Data":"6c44d8242199bb11f7d578d41d5b29ab3a5dcab8d89e5c506f20647b7852e054"} Nov 01 07:14:41.710505 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.710489 4914 scope.go:122] "RemoveContainer" containerID="1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de" Nov 01 07:14:41.710793 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:41.710771 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de\": container with ID starting with 1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de not found: ID does not exist" containerID="1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de" Nov 01 07:14:41.710871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.710803 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de"} err="failed to get container status \"1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de\": rpc error: code = NotFound desc = could not find container \"1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de\": container with ID starting with 1a810943ea756f09e5a059b6c0a4664c55e1392dab2d0dcbbe362241a9f616de not found: ID does not exist" Nov 01 07:14:41.710871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.710823 4914 scope.go:122] "RemoveContainer" containerID="0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99" Nov 01 07:14:41.720560 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.720537 4914 scope.go:122] "RemoveContainer" containerID="0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99" Nov 01 07:14:41.720811 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:41.720793 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99\": container with ID starting with 0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99 not found: ID does not exist" containerID="0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99" Nov 01 07:14:41.720877 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.720819 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99"} err="failed to get container status \"0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99\": rpc error: code = NotFound desc = could not find container \"0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99\": container with ID starting with 0524a3005085a875cb321d9948d945dabd865a28e574fca1783a74bf71d49b99 not found: ID does not exist" Nov 01 07:14:41.862469 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.862440 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct"] Nov 01 07:14:41.868559 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.868525 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-51aa8f-5968bc585c-vt9ct"] Nov 01 07:14:41.977241 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.977155 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg"] Nov 01 07:14:41.979703 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:41.979678 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-51aa8f-56555b749f-z9fxg"] Nov 01 07:14:42.615745 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:42.615708 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" path="/var/lib/kubelet/pods/b4f27112-de78-4352-bee3-b1533e827e0d/volumes" Nov 01 07:14:42.616114 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:42.616098 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" path="/var/lib/kubelet/pods/f90e61ce-cc69-4f71-b18a-23b8da40aeab/volumes" Nov 01 07:14:42.717668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:42.717629 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" event={"ID":"1d2cc16e-c9b1-4e60-b632-c824124630c7","Type":"ContainerStarted","Data":"06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc"} Nov 01 07:14:42.835711 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:42.835661 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podStartSLOduration=2.835648201 podStartE2EDuration="2.835648201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:14:42.833517829 +0000 UTC m=+3898.386307024" watchObservedRunningTime="2026-11-01 07:14:42.835648201 +0000 UTC m=+3898.388437395" Nov 01 07:14:43.215867 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:43.215837 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:43.220838 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:43.220810 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:43.722592 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:43.722551 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:43.723464 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:43.723444 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:14:45.111967 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:45.111937 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:14:45.125998 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:45.125977 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:14:49.139579 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.139549 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt"] Nov 01 07:14:49.139968 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.139958 4914 cpu_manager.go:414] "containerMap: removing container" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerName="payload-pre-processing" Nov 01 07:14:49.140011 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.139970 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerName="payload-pre-processing" Nov 01 07:14:49.140011 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.139983 4914 cpu_manager.go:414] "containerMap: removing container" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" containerName="payload-processing" Nov 01 07:14:49.140011 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.139988 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" containerName="payload-processing" Nov 01 07:14:49.140103 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.140048 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90e61ce-cc69-4f71-b18a-23b8da40aeab" containerName="payload-pre-processing" Nov 01 07:14:49.140103 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.140077 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f27112-de78-4352-bee3-b1533e827e0d" containerName="payload-processing" Nov 01 07:14:49.144827 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.144809 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.185649 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.185612 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrn4c\" (UniqueName: \"kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c\") pod \"payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.185766 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.185723 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.198609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.198352 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-main-976689-dockercfg-sj52j\"" Nov 01 07:14:49.198609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.198454 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-main-976689\"" Nov 01 07:14:49.199955 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.199932 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt"] Nov 01 07:14:49.286915 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.286875 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn4c\" (UniqueName: \"kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c\") pod \"payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.287116 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.286956 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.287784 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.287759 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.295544 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.295514 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4"] Nov 01 07:14:49.299490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.299472 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.342926 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.342902 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn4c\" (UniqueName: \"kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c\") pod \"payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.356050 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.356024 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4"] Nov 01 07:14:49.387422 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.387396 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxvx\" (UniqueName: \"kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx\") pod \"payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.387530 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.387455 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.457226 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.457172 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:49.488385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.488354 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.488497 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.488461 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxvx\" (UniqueName: \"kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx\") pod \"payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.489111 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.489092 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume\") pod \"payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.544546 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.544523 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxvx\" (UniqueName: \"kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx\") pod \"payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.612113 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.612087 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:49.825091 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.825070 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt"] Nov 01 07:14:49.827338 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:14:49.827304 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3640743c_d26d_4194_9ac7_2ecdd08f88df.slice/crio-ff9efc1cff7740a1ceb54d5f743f971d119d32beb8a3aa700b6f20152ffb28b4 WatchSource:0}: Error finding container ff9efc1cff7740a1ceb54d5f743f971d119d32beb8a3aa700b6f20152ffb28b4: Status 404 returned error can't find the container with id ff9efc1cff7740a1ceb54d5f743f971d119d32beb8a3aa700b6f20152ffb28b4 Nov 01 07:14:49.981300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:49.981250 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4"] Nov 01 07:14:49.982789 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:14:49.982766 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9c9eaf_c1df_4b22_b791_3ef4ab359e03.slice/crio-b1edcc9922ee5d3412e149c3554101c7ec6700995e967e187ab95a49a87b870e WatchSource:0}: Error finding container b1edcc9922ee5d3412e149c3554101c7ec6700995e967e187ab95a49a87b870e: Status 404 returned error can't find the container with id b1edcc9922ee5d3412e149c3554101c7ec6700995e967e187ab95a49a87b870e Nov 01 07:14:50.753592 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.753561 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" event={"ID":"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03","Type":"ContainerStarted","Data":"80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095"} Nov 01 07:14:50.754038 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.753611 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" event={"ID":"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03","Type":"ContainerStarted","Data":"b1edcc9922ee5d3412e149c3554101c7ec6700995e967e187ab95a49a87b870e"} Nov 01 07:14:50.754038 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.753667 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:14:50.755146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.755124 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" event={"ID":"3640743c-d26d-4194-9ac7-2ecdd08f88df","Type":"ContainerStarted","Data":"eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669"} Nov 01 07:14:50.755199 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.755168 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" event={"ID":"3640743c-d26d-4194-9ac7-2ecdd08f88df","Type":"ContainerStarted","Data":"ff9efc1cff7740a1ceb54d5f743f971d119d32beb8a3aa700b6f20152ffb28b4"} Nov 01 07:14:50.755245 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.755210 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:14:50.866177 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.866135 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" podStartSLOduration=1.866123521 podStartE2EDuration="1.866123521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:14:50.865855379 +0000 UTC m=+3906.418644572" watchObservedRunningTime="2026-11-01 07:14:50.866123521 +0000 UTC m=+3906.418912728" Nov 01 07:14:50.992368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:50.992300 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" podStartSLOduration=1.992262056 podStartE2EDuration="1.992262056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:14:50.989136431 +0000 UTC m=+3906.541925627" watchObservedRunningTime="2026-11-01 07:14:50.992262056 +0000 UTC m=+3906.545051252" Nov 01 07:14:51.671125 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.671089 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r"] Nov 01 07:14:51.674430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.674411 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.725888 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.725861 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-main-976689-serving-cert\"" Nov 01 07:14:51.777290 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.777253 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r"] Nov 01 07:14:51.809997 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.809968 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcxc\" (UniqueName: \"kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.810124 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.810017 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.810259 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.810236 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.810362 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.810327 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.920319 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.920032 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.920319 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.920092 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.920319 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.920145 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcxc\" (UniqueName: \"kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.920319 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.920176 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.922235 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:51.920815 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-serving-cert: secret "maas-api-e2e-worker-main-976689-serving-cert" not found Nov 01 07:14:51.922235 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:14:51.920903 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls podName:a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c nodeName:}" failed. No retries permitted until 2026-11-01 07:14:52.420882743 +0000 UTC m=+3907.973671923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls") pod "maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" (UID: "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c") : secret "maas-api-e2e-worker-main-976689-serving-cert" not found Nov 01 07:14:51.922889 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.922844 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.933094 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.933062 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:51.979446 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:51.979417 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcxc\" (UniqueName: \"kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:52.427679 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:52.427645 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:52.430354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:52.430329 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\") pod \"maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:52.586959 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:52.586932 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:52.956853 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:52.956828 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r"] Nov 01 07:14:52.959100 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:14:52.959075 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f082c7_47c0_4aa6_a19e_5aa4b5c5ad1c.slice/crio-3b8d577bfae4783b19f27d7a3fa5dc2e24e929a5c73f5d36dcdfabd9a3dfe919 WatchSource:0}: Error finding container 3b8d577bfae4783b19f27d7a3fa5dc2e24e929a5c73f5d36dcdfabd9a3dfe919: Status 404 returned error can't find the container with id 3b8d577bfae4783b19f27d7a3fa5dc2e24e929a5c73f5d36dcdfabd9a3dfe919 Nov 01 07:14:52.963997 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:52.963980 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:14:53.771611 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:53.771564 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" event={"ID":"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c","Type":"ContainerStarted","Data":"3b8d577bfae4783b19f27d7a3fa5dc2e24e929a5c73f5d36dcdfabd9a3dfe919"} Nov 01 07:14:54.777945 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:54.777909 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" event={"ID":"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c","Type":"ContainerStarted","Data":"763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e"} Nov 01 07:14:54.778372 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:54.778014 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:14:54.891830 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:14:54.891781 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" podStartSLOduration=3.044396892 podStartE2EDuration="3.891768907s" totalImagesPullingTime="847.372015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:14:54.891024016 +0000 UTC m=+3910.443813208" watchObservedRunningTime="2026-11-01 07:14:54.891768907 +0000 UTC m=+3910.444558100" Nov 01 07:15:00.788323 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:00.788221 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:15:01.772871 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:01.772831 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:15:01.773057 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:01.772903 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:15:07.592090 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:15:07.592051 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:15:08.857151 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:08.857123 4914 generic.go:356] "Generic (PLEG): container finished" podID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerID="06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc" exitCode=0 Nov 01 07:15:08.857751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:08.857182 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" event={"ID":"1d2cc16e-c9b1-4e60-b632-c824124630c7","Type":"ContainerDied","Data":"06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc"} Nov 01 07:15:08.857751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:08.857497 4914 scope.go:122] "RemoveContainer" containerID="06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc" Nov 01 07:15:08.858619 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:08.858590 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:15:08.858686 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:08.858662 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:15:08.858738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:08.858697 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:15:09.875971 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:09.875931 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" event={"ID":"1d2cc16e-c9b1-4e60-b632-c824124630c7","Type":"ContainerStarted","Data":"f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb"} Nov 01 07:15:10.216981 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:10.216867 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:15:10.218206 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:10.218177 4914 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-mgddl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 07:15:10.218355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:10.218239 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 07:15:11.216049 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:11.216013 4914 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-mgddl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 07:15:11.216502 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:11.216076 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 07:15:12.216837 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:12.216806 4914 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-mgddl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 07:15:12.217240 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:12.216864 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 07:15:13.216055 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:13.216021 4914 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-mgddl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 07:15:13.216227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:13.216076 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 07:15:13.722443 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:13.722402 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:15:14.216869 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:14.216825 4914 patch_prober.go:29] interesting pod/e2e-worker-main-976689-openshift-default-5db554f479-mgddl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 01 07:15:14.217062 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:14.216917 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 01 07:15:15.219787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:15.219757 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:15:15.220785 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:15.220762 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:15:19.918207 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:19.918175 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44"] Nov 01 07:15:19.918721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:19.918497 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" containerID="cri-o://8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d" gracePeriod=30 Nov 01 07:15:19.969146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:19.969109 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn"] Nov 01 07:15:19.969404 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:19.969378 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" containerID="cri-o://919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65" gracePeriod=30 Nov 01 07:15:25.218497 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.218469 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:15:25.254925 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.254904 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:15:25.282523 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282503 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282620 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282535 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282620 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282570 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282720 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282629 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282720 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282661 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282720 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282712 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282881 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282763 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282881 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282801 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.282881 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282862 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l\") pod \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\" (UID: \"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b\") " Nov 01 07:15:25.283046 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.282921 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.283107 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283056 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.283162 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283135 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.283162 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283151 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.283306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283237 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.283306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283258 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.283306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283302 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.283439 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283317 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.283439 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.283351 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:15:25.284738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.284719 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.285116 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.285100 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "kube-api-access-7sr4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:25.285411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.285397 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:25.285473 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.285422 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo" pod "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" (UID: "fbd2cbe8-495b-4c88-9e09-a439ae9fc60b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:15:25.384206 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384182 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384217 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384239 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384261 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384354 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384318 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384366 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384399 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384430 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384488 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data\") pod \"1d08809f-cdc5-4f43-b478-7106de8beedb\" (UID: \"1d08809f-cdc5-4f43-b478-7106de8beedb\") " Nov 01 07:15:25.384599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384526 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.384909 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384743 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.384909 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384672 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:15:25.385025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384944 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.385025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.384753 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385024 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385045 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d08809f-cdc5-4f43-b478-7106de8beedb-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385060 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385075 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sr4l\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-kube-api-access-7sr4l\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385092 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385107 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385120 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.385138 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.385134 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.386724 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.386692 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:15:25.387786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.387762 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "kube-api-access-kzqjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:25.388355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.388334 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:15:25.388946 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.388929 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token" pod "1d08809f-cdc5-4f43-b478-7106de8beedb" (UID: "1d08809f-cdc5-4f43-b478-7106de8beedb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:15:25.485937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.485883 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.485937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.485906 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.485937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.485917 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.485937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.485927 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.486140 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.485943 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzqjq\" (UniqueName: \"kubernetes.io/projected/1d08809f-cdc5-4f43-b478-7106de8beedb-kube-api-access-kzqjq\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.486140 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.485952 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d08809f-cdc5-4f43-b478-7106de8beedb-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:15:25.956164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.956135 4914 generic.go:356] "Generic (PLEG): container finished" podID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerID="8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d" exitCode=0 Nov 01 07:15:25.956337 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.956188 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" Nov 01 07:15:25.956337 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.956212 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" event={"ID":"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b","Type":"ContainerDied","Data":"8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d"} Nov 01 07:15:25.956337 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.956261 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44" event={"ID":"fbd2cbe8-495b-4c88-9e09-a439ae9fc60b","Type":"ContainerDied","Data":"86f0187088e63fafeca12a5ce93fc293ec09b53e4232f683d10aecd45927ffa8"} Nov 01 07:15:25.956337 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.956312 4914 scope.go:122] "RemoveContainer" containerID="8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d" Nov 01 07:15:25.957716 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.957692 4914 generic.go:356] "Generic (PLEG): container finished" podID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerID="919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65" exitCode=0 Nov 01 07:15:25.957856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.957741 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" Nov 01 07:15:25.957856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.957776 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" event={"ID":"1d08809f-cdc5-4f43-b478-7106de8beedb","Type":"ContainerDied","Data":"919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65"} Nov 01 07:15:25.957856 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.957819 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn" event={"ID":"1d08809f-cdc5-4f43-b478-7106de8beedb","Type":"ContainerDied","Data":"77ee161b2f0154bde1cdf2fc132f285d3fe4b37ec881c3ae0c00813e33cf6f27"} Nov 01 07:15:25.966999 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.966982 4914 scope.go:122] "RemoveContainer" containerID="8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d" Nov 01 07:15:25.967332 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:15:25.967305 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d\": container with ID starting with 8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d not found: ID does not exist" containerID="8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d" Nov 01 07:15:25.967418 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.967341 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d"} err="failed to get container status \"8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d\": rpc error: code = NotFound desc = could not find container \"8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d\": container with ID starting with 8115ac99bd3b027f00f6d45994cfd2ec16eb89141aa36f0a76445e047390d11d not found: ID does not exist" Nov 01 07:15:25.967418 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.967359 4914 scope.go:122] "RemoveContainer" containerID="919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65" Nov 01 07:15:25.976028 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.976004 4914 scope.go:122] "RemoveContainer" containerID="919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65" Nov 01 07:15:25.976302 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:15:25.976261 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65\": container with ID starting with 919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65 not found: ID does not exist" containerID="919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65" Nov 01 07:15:25.976425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:25.976306 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65"} err="failed to get container status \"919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65\": rpc error: code = NotFound desc = could not find container \"919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65\": container with ID starting with 919831c5636ac6f453f781de28c26574494046784f115d7be40eda14f1314a65 not found: ID does not exist" Nov 01 07:15:26.122051 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:26.122026 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn"] Nov 01 07:15:26.171248 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:26.171223 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-7x5qn"] Nov 01 07:15:26.233467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:26.233404 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44"] Nov 01 07:15:26.237190 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:26.237167 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9hv44"] Nov 01 07:15:26.615356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:26.615320 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" path="/var/lib/kubelet/pods/1d08809f-cdc5-4f43-b478-7106de8beedb/volumes" Nov 01 07:15:26.615783 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:15:26.615769 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" path="/var/lib/kubelet/pods/fbd2cbe8-495b-4c88-9e09-a439ae9fc60b/volumes" Nov 01 07:15:37.594130 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:15:37.594081 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:16:07.591663 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:16:07.591621 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:16:37.591772 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:16:37.591684 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:16:50.626483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626446 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87"] Nov 01 07:16:50.626994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626829 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" Nov 01 07:16:50.626994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626840 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" Nov 01 07:16:50.626994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626865 4914 cpu_manager.go:414] "containerMap: removing container" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" Nov 01 07:16:50.626994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626870 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" Nov 01 07:16:50.626994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626943 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d08809f-cdc5-4f43-b478-7106de8beedb" containerName="istio-proxy" Nov 01 07:16:50.626994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.626953 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd2cbe8-495b-4c88-9e09-a439ae9fc60b" containerName="istio-proxy" Nov 01 07:16:50.630124 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.630102 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808241 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q7gq\" (UniqueName: \"kubernetes.io/projected/430c5975-a53c-4dde-814a-6661ad4e71b5-kube-api-access-6q7gq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808454 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808298 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808454 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808386 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808454 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808435 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808520 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430c5975-a53c-4dde-814a-6661ad4e71b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808552 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808575 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808568 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808698 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808610 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.808698 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.808669 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.833505 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.833476 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87"] Nov 01 07:16:50.910119 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910047 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910119 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910096 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430c5975-a53c-4dde-814a-6661ad4e71b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910125 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910143 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910165 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910193 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910246 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7gq\" (UniqueName: \"kubernetes.io/projected/430c5975-a53c-4dde-814a-6661ad4e71b5-kube-api-access-6q7gq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910316 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910368 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.910847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910821 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.911195 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910923 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.911195 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.910999 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.911347 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.911036 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430c5975-a53c-4dde-814a-6661ad4e71b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.911388 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.911327 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.912939 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.912919 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.913034 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.912970 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.966424 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.966393 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430c5975-a53c-4dde-814a-6661ad4e71b5-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:50.966692 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:50.966670 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q7gq\" (UniqueName: \"kubernetes.io/projected/430c5975-a53c-4dde-814a-6661ad4e71b5-kube-api-access-6q7gq\") pod \"maas-default-gateway-openshift-default-7d9897b54c-krv87\" (UID: \"430c5975-a53c-4dde-814a-6661ad4e71b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:51.243022 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:51.242950 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:51.661758 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:51.661735 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87"] Nov 01 07:16:51.664230 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:16:51.664194 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430c5975_a53c_4dde_814a_6661ad4e71b5.slice/crio-a0288345c4ea1138aa63d1effb881aae14c1d40c9bbbd8a6158944d99fe2a4b0 WatchSource:0}: Error finding container a0288345c4ea1138aa63d1effb881aae14c1d40c9bbbd8a6158944d99fe2a4b0: Status 404 returned error can't find the container with id a0288345c4ea1138aa63d1effb881aae14c1d40c9bbbd8a6158944d99fe2a4b0 Nov 01 07:16:51.666385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:51.666353 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:16:51.666467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:51.666415 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:16:51.666467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:51.666447 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:16:52.375615 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:52.375572 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" event={"ID":"430c5975-a53c-4dde-814a-6661ad4e71b5","Type":"ContainerStarted","Data":"1d69bdc2e511236e102626911d419ffae1eebca4476ad3a38f4382b3532059e1"} Nov 01 07:16:52.375615 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:52.375622 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" event={"ID":"430c5975-a53c-4dde-814a-6661ad4e71b5","Type":"ContainerStarted","Data":"a0288345c4ea1138aa63d1effb881aae14c1d40c9bbbd8a6158944d99fe2a4b0"} Nov 01 07:16:52.591812 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:52.591753 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podStartSLOduration=2.5917393349999998 podStartE2EDuration="2.591739335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:16:52.541086015 +0000 UTC m=+4028.093875210" watchObservedRunningTime="2026-11-01 07:16:52.591739335 +0000 UTC m=+4028.144528528" Nov 01 07:16:53.243850 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:53.243817 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:16:53.245182 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:53.245150 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 07:16:53.245334 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:53.245196 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 07:16:54.243765 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:54.243726 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 07:16:54.243966 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:54.243783 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 07:16:55.243944 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:55.243913 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 07:16:55.244122 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:55.243962 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 07:16:56.243724 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:56.243685 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 07:16:56.243901 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:56.243746 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 07:16:57.244145 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:57.244107 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 07:16:57.244591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:57.244174 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 07:16:58.244212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:58.244179 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 01 07:16:58.244622 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:16:58.244228 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 01 07:17:00.244114 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:00.244076 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:17:00.244557 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:00.244140 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:17:01.285533 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:01.285481 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:17:01.286770 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:01.286258 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:17:02.007396 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:02.007360 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:17:02.007779 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:02.007758 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:17:02.008478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:02.008455 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" Nov 01 07:17:05.725732 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.725697 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8"] Nov 01 07:17:05.729251 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.729234 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772089 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772056 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0dea1e-43ae-409a-9eeb-02badf63542c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772207 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772101 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772202 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772240 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772352 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772255 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772352 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772315 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772445 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772426 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772509 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772496 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.772585 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.772548 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsk7\" (UniqueName: \"kubernetes.io/projected/3f0dea1e-43ae-409a-9eeb-02badf63542c-kube-api-access-gmsk7\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.832835 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.832797 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8"] Nov 01 07:17:05.873545 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873523 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsk7\" (UniqueName: \"kubernetes.io/projected/3f0dea1e-43ae-409a-9eeb-02badf63542c-kube-api-access-gmsk7\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873683 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873553 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0dea1e-43ae-409a-9eeb-02badf63542c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873683 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873572 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873683 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873619 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873683 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873658 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873891 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873682 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873891 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873723 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873891 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873758 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.873891 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.873833 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.874159 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.874136 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.874253 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.874228 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.874336 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.874321 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.874381 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.874343 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0dea1e-43ae-409a-9eeb-02badf63542c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.874453 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.874427 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.876154 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.876132 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.876386 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.876369 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.929315 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.929270 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0dea1e-43ae-409a-9eeb-02badf63542c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:05.929826 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:05.929805 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsk7\" (UniqueName: \"kubernetes.io/projected/3f0dea1e-43ae-409a-9eeb-02badf63542c-kube-api-access-gmsk7\") pod \"maas-default-gateway-openshift-default-7d9897b54c-vktw8\" (UID: \"3f0dea1e-43ae-409a-9eeb-02badf63542c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:06.042425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:06.042367 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:06.475218 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:06.475192 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8"] Nov 01 07:17:06.477520 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:17:06.477487 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0dea1e_43ae_409a_9eeb_02badf63542c.slice/crio-8b889b23514d1c19d7dc65d27f2d9a403d328d842f6eb133e7aca6faf9c13eff WatchSource:0}: Error finding container 8b889b23514d1c19d7dc65d27f2d9a403d328d842f6eb133e7aca6faf9c13eff: Status 404 returned error can't find the container with id 8b889b23514d1c19d7dc65d27f2d9a403d328d842f6eb133e7aca6faf9c13eff Nov 01 07:17:06.479580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:06.479550 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:17:06.479639 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:06.479617 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:17:06.479681 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:06.479646 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:17:07.472747 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:07.472689 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" event={"ID":"3f0dea1e-43ae-409a-9eeb-02badf63542c","Type":"ContainerStarted","Data":"d6eba4bb29a9a3b904b06c4e93a1834a8b746c755114b49cb7b1de793c7acabd"} Nov 01 07:17:07.473250 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:07.472800 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" event={"ID":"3f0dea1e-43ae-409a-9eeb-02badf63542c","Type":"ContainerStarted","Data":"8b889b23514d1c19d7dc65d27f2d9a403d328d842f6eb133e7aca6faf9c13eff"} Nov 01 07:17:07.598022 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:17:07.597978 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:17:07.637936 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:07.637878 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podStartSLOduration=2.6378638949999997 podStartE2EDuration="2.637863895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:17:07.635603995 +0000 UTC m=+4043.188393188" watchObservedRunningTime="2026-11-01 07:17:07.637863895 +0000 UTC m=+4043.190653150" Nov 01 07:17:08.043500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:08.043467 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:08.044782 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:08.044753 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 07:17:08.044905 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:08.044826 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 07:17:09.043146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:09.043109 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 07:17:09.043602 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:09.043164 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 07:17:10.043453 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:10.043417 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 07:17:10.043858 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:10.043481 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 07:17:11.043294 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:11.043241 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 07:17:11.043483 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:11.043341 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 07:17:12.043156 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:12.043113 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 07:17:12.043400 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:12.043163 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 07:17:13.043110 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:13.043075 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 01 07:17:13.043498 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:13.043137 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 01 07:17:14.671653 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:14.671618 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:14.672117 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:14.671966 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:14.672919 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:17:14.672896 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" Nov 01 07:17:37.591589 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:17:37.591553 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:18:07.590911 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:18:07.590829 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:18:37.591569 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:18:37.591523 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:19:07.593138 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:19:07.593100 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:19:13.823486 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:13.823443 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:19:13.824017 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:13.823516 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:19:21.757302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.757256 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28"] Nov 01 07:19:21.760950 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.760932 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796097 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796073 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796108 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796136 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796190 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796393 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796304 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796393 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796354 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76rc\" (UniqueName: \"kubernetes.io/projected/c97e7d90-1e4f-4445-84e6-6198f8bd3024-kube-api-access-c76rc\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796499 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796438 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796546 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796524 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.796586 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.796552 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.864908 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.864884 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28"] Nov 01 07:19:21.897518 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897494 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897634 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897556 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897634 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897580 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897634 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897603 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897635 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897662 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897690 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897720 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.897787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.897756 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76rc\" (UniqueName: \"kubernetes.io/projected/c97e7d90-1e4f-4445-84e6-6198f8bd3024-kube-api-access-c76rc\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.898108 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.898077 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.898176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.898156 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.898474 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.898395 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.898660 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.898528 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.898660 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.898628 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.899814 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.899790 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.900175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.900156 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.953964 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.953942 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97e7d90-1e4f-4445-84e6-6198f8bd3024-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:21.954088 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:21.954067 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76rc\" (UniqueName: \"kubernetes.io/projected/c97e7d90-1e4f-4445-84e6-6198f8bd3024-kube-api-access-c76rc\") pod \"maas-default-gateway-openshift-default-7d9897b54c-9tp28\" (UID: \"c97e7d90-1e4f-4445-84e6-6198f8bd3024\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:22.074597 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:22.074538 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:22.495176 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:22.495152 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28"] Nov 01 07:19:22.496994 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:19:22.496966 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97e7d90_1e4f_4445_84e6_6198f8bd3024.slice/crio-50f9ed037873e310552a9fecce1f6c63d9a1544d3e07711e471fdf1b99e4c354 WatchSource:0}: Error finding container 50f9ed037873e310552a9fecce1f6c63d9a1544d3e07711e471fdf1b99e4c354: Status 404 returned error can't find the container with id 50f9ed037873e310552a9fecce1f6c63d9a1544d3e07711e471fdf1b99e4c354 Nov 01 07:19:22.499105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:22.499075 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:19:22.499172 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:22.499143 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:19:22.499210 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:22.499174 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:19:23.139360 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:23.139313 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" event={"ID":"c97e7d90-1e4f-4445-84e6-6198f8bd3024","Type":"ContainerStarted","Data":"d43a5e08b598586949fab7042526c82e8a592400bad527754e653ae2d90440d8"} Nov 01 07:19:23.139834 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:23.139382 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" event={"ID":"c97e7d90-1e4f-4445-84e6-6198f8bd3024","Type":"ContainerStarted","Data":"50f9ed037873e310552a9fecce1f6c63d9a1544d3e07711e471fdf1b99e4c354"} Nov 01 07:19:23.306395 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:23.306334 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podStartSLOduration=2.306313931 podStartE2EDuration="2.306313931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:19:23.30502607 +0000 UTC m=+4178.857815286" watchObservedRunningTime="2026-11-01 07:19:23.306313931 +0000 UTC m=+4178.859103128" Nov 01 07:19:24.075559 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:24.075522 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:24.077110 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:24.077088 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 07:19:24.077224 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:24.077153 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 07:19:25.075749 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:25.075713 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 07:19:25.076212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:25.075768 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 07:19:26.075785 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:26.075709 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 07:19:26.075785 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:26.075763 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 07:19:27.075765 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:27.075730 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 07:19:27.075956 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:27.075781 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 07:19:28.075769 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:28.075731 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 07:19:28.075980 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:28.075784 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 07:19:29.075460 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:29.075409 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Nov 01 07:19:29.075644 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:29.075460 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Nov 01 07:19:30.736145 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:30.736115 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:30.736680 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:30.736655 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:30.737373 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:30.737352 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" Nov 01 07:19:37.593087 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:19:37.593051 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:19:45.144984 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:45.144955 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:19:45.158621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:19:45.158590 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:20:07.715306 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:20:07.713635 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:20:31.796429 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:20:31.796395 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:20:31.797031 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:20:31.796528 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:20:37.591786 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:20:37.591752 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:21:01.785370 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:21:01.783178 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:21:01.785370 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:21:01.783256 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:21:07.591236 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:21:07.591200 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:21:37.591241 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:21:37.591206 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:21:45.728459 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:21:45.728402 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:21:45.729212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:21:45.729160 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:22:07.705627 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:22:07.705580 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:22:18.053302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:18.052382 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:22:18.053302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:18.052454 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:22:37.712354 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:22:37.708983 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:22:49.942581 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:49.942548 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4"] Nov 01 07:22:50.050622 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:50.050569 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt"] Nov 01 07:22:54.943163 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:54.943118 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" podUID="2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" containerName="payload-processing" containerID="cri-o://80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095" gracePeriod=25 Nov 01 07:22:55.051839 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.051782 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" podUID="3640743c-d26d-4194-9ac7-2ecdd08f88df" containerName="payload-pre-processing" containerID="cri-o://eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669" gracePeriod=25 Nov 01 07:22:55.192145 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.192120 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:22:55.250265 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.250230 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume\") pod \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " Nov 01 07:22:55.250485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.250330 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx\" (UniqueName: \"kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx\") pod \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\" (UID: \"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03\") " Nov 01 07:22:55.250751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.250723 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume" pod "2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" (UID: "2b9c9eaf-c1df-4b22-b791-3ef4ab359e03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:22:55.252759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.252729 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx" pod "2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" (UID: "2b9c9eaf-c1df-4b22-b791-3ef4ab359e03"). InnerVolumeSpecName "kube-api-access-ngxvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:22:55.293954 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.293928 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:22:55.351040 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.351010 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c\" (UniqueName: \"kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c\") pod \"3640743c-d26d-4194-9ac7-2ecdd08f88df\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " Nov 01 07:22:55.351211 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.351163 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume\") pod \"3640743c-d26d-4194-9ac7-2ecdd08f88df\" (UID: \"3640743c-d26d-4194-9ac7-2ecdd08f88df\") " Nov 01 07:22:55.351360 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.351347 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:55.351416 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.351362 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngxvx\" (UniqueName: \"kubernetes.io/projected/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03-kube-api-access-ngxvx\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:55.351566 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.351543 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume" pod "3640743c-d26d-4194-9ac7-2ecdd08f88df" (UID: "3640743c-d26d-4194-9ac7-2ecdd08f88df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:22:55.353196 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.353174 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c" pod "3640743c-d26d-4194-9ac7-2ecdd08f88df" (UID: "3640743c-d26d-4194-9ac7-2ecdd08f88df"). InnerVolumeSpecName "kube-api-access-mrn4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:22:55.452489 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.452451 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3640743c-d26d-4194-9ac7-2ecdd08f88df-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:55.452489 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.452480 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrn4c\" (UniqueName: \"kubernetes.io/projected/3640743c-d26d-4194-9ac7-2ecdd08f88df-kube-api-access-mrn4c\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:22:55.684625 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.684591 4914 generic.go:356] "Generic (PLEG): container finished" podID="2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" containerID="80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095" exitCode=0 Nov 01 07:22:55.684843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.684636 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" Nov 01 07:22:55.684843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.684681 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" event={"ID":"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03","Type":"ContainerDied","Data":"80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095"} Nov 01 07:22:55.684843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.684736 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4" event={"ID":"2b9c9eaf-c1df-4b22-b791-3ef4ab359e03","Type":"ContainerDied","Data":"b1edcc9922ee5d3412e149c3554101c7ec6700995e967e187ab95a49a87b870e"} Nov 01 07:22:55.684843 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.684764 4914 scope.go:122] "RemoveContainer" containerID="80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095" Nov 01 07:22:55.686117 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.686097 4914 generic.go:356] "Generic (PLEG): container finished" podID="3640743c-d26d-4194-9ac7-2ecdd08f88df" containerID="eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669" exitCode=0 Nov 01 07:22:55.686184 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.686154 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" Nov 01 07:22:55.686184 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.686167 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" event={"ID":"3640743c-d26d-4194-9ac7-2ecdd08f88df","Type":"ContainerDied","Data":"eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669"} Nov 01 07:22:55.686302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.686207 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt" event={"ID":"3640743c-d26d-4194-9ac7-2ecdd08f88df","Type":"ContainerDied","Data":"ff9efc1cff7740a1ceb54d5f743f971d119d32beb8a3aa700b6f20152ffb28b4"} Nov 01 07:22:55.695065 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.695043 4914 scope.go:122] "RemoveContainer" containerID="80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095" Nov 01 07:22:55.695381 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:22:55.695361 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095\": container with ID starting with 80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095 not found: ID does not exist" containerID="80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095" Nov 01 07:22:55.695482 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.695394 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095"} err="failed to get container status \"80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095\": rpc error: code = NotFound desc = could not find container \"80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095\": container with ID starting with 80987d6d9ad55120fab0ea555158e0ae9640e82c765988481801170f94ee4095 not found: ID does not exist" Nov 01 07:22:55.695482 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.695411 4914 scope.go:122] "RemoveContainer" containerID="eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669" Nov 01 07:22:55.703300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.703253 4914 scope.go:122] "RemoveContainer" containerID="eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669" Nov 01 07:22:55.703581 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:22:55.703562 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669\": container with ID starting with eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669 not found: ID does not exist" containerID="eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669" Nov 01 07:22:55.703648 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.703592 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669"} err="failed to get container status \"eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669\": rpc error: code = NotFound desc = could not find container \"eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669\": container with ID starting with eaa34d1c91118667d4b4de89b18f3d51db7b56c7a8e99004e29ef7201a1b4669 not found: ID does not exist" Nov 01 07:22:55.845532 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.845475 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt"] Nov 01 07:22:55.847123 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.847102 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-main-976689-68f8fb7887-qqdnt"] Nov 01 07:22:55.956633 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.956545 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4"] Nov 01 07:22:55.962436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:55.962411 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-main-976689-848c4986d9-x4sx4"] Nov 01 07:22:56.615269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:56.615226 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" path="/var/lib/kubelet/pods/2b9c9eaf-c1df-4b22-b791-3ef4ab359e03/volumes" Nov 01 07:22:56.615731 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:22:56.615711 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3640743c-d26d-4194-9ac7-2ecdd08f88df" path="/var/lib/kubelet/pods/3640743c-d26d-4194-9ac7-2ecdd08f88df/volumes" Nov 01 07:23:02.671562 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:02.671526 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r"] Nov 01 07:23:02.672015 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:02.671743 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" podUID="a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" containerName="maas-api" containerID="cri-o://763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e" gracePeriod=30 Nov 01 07:23:02.955882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:02.955855 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:23:03.027766 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.027738 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc\" (UniqueName: \"kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc\") pod \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " Nov 01 07:23:03.027942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.027771 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca\") pod \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " Nov 01 07:23:03.027942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.027924 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\") pod \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " Nov 01 07:23:03.028026 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.027977 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs\") pod \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\" (UID: \"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c\") " Nov 01 07:23:03.028223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.028196 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca" pod "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" (UID: "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:23:03.028313 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.028263 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:03.030051 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.030028 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls" pod "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" (UID: "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:23:03.030161 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.030144 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc" pod "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" (UID: "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c"). InnerVolumeSpecName "kube-api-access-8hcxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:23:03.030211 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.030166 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs" pod "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" (UID: "a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:23:03.129720 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.129687 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:03.129720 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.129717 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:03.129928 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.129727 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hcxc\" (UniqueName: \"kubernetes.io/projected/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c-kube-api-access-8hcxc\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:03.727260 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.727225 4914 generic.go:356] "Generic (PLEG): container finished" podID="a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" containerID="763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e" exitCode=0 Nov 01 07:23:03.727674 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.727287 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" Nov 01 07:23:03.727674 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.727319 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" event={"ID":"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c","Type":"ContainerDied","Data":"763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e"} Nov 01 07:23:03.727674 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.727376 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r" event={"ID":"a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c","Type":"ContainerDied","Data":"3b8d577bfae4783b19f27d7a3fa5dc2e24e929a5c73f5d36dcdfabd9a3dfe919"} Nov 01 07:23:03.727674 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.727400 4914 scope.go:122] "RemoveContainer" containerID="763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e" Nov 01 07:23:03.737153 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.737131 4914 scope.go:122] "RemoveContainer" containerID="763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e" Nov 01 07:23:03.737574 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:03.737553 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e\": container with ID starting with 763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e not found: ID does not exist" containerID="763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e" Nov 01 07:23:03.737629 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.737583 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e"} err="failed to get container status \"763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e\": rpc error: code = NotFound desc = could not find container \"763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e\": container with ID starting with 763336322aff5897e9e56665c98768a9a1f8390ea58910a4602cfd8ffc06963e not found: ID does not exist" Nov 01 07:23:03.891207 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.891174 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r"] Nov 01 07:23:03.893102 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:03.893074 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-main-976689-7b8f4d7f4f-8s97r"] Nov 01 07:23:04.615209 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:04.615165 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" path="/var/lib/kubelet/pods/a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c/volumes" Nov 01 07:23:07.596054 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:07.596015 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:23:21.018378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:21.018346 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl"] Nov 01 07:23:21.018876 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:21.018594 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" containerID="cri-o://f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb" gracePeriod=30 Nov 01 07:23:25.751693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.751659 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44"] Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752052 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" containerName="payload-processing" Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752064 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" containerName="payload-processing" Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752090 4914 cpu_manager.go:414] "containerMap: removing container" podUID="a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" containerName="maas-api" Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752096 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" containerName="maas-api" Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752105 4914 cpu_manager.go:414] "containerMap: removing container" podUID="3640743c-d26d-4194-9ac7-2ecdd08f88df" containerName="payload-pre-processing" Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752110 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3640743c-d26d-4194-9ac7-2ecdd08f88df" containerName="payload-pre-processing" Nov 01 07:23:25.752181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752179 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c9eaf-c1df-4b22-b791-3ef4ab359e03" containerName="payload-processing" Nov 01 07:23:25.752495 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752192 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f082c7-47c0-4aa6-a19e-5aa4b5c5ad1c" containerName="maas-api" Nov 01 07:23:25.752495 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.752198 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3640743c-d26d-4194-9ac7-2ecdd08f88df" containerName="payload-pre-processing" Nov 01 07:23:25.755347 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.755325 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855222 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855185 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855400 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855229 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855400 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855262 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855400 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855308 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855400 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855369 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855401 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855434 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855541 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.855627 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.855568 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhj5\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.858927 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.858898 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44"] Nov 01 07:23:25.861012 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.860978 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-501c44d6-openshift-default-dockercfg-lldv4\"" Nov 01 07:23:25.959820 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.959782 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.959888 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.959913 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhj5\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960256 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960231 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960488 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960462 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960255 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960506 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960512 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960621 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960263 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960848 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960638 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960848 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960707 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.960848 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.960747 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.961210 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.961110 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.961568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.961546 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.963852 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.963826 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:25.965985 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:25.965966 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:26.017337 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.017240 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhj5\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:26.017584 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.017567 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token\") pod \"e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:26.068974 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.068939 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:26.261490 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.261463 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:23:26.364539 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364511 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364567 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364588 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364645 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364692 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364947 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364798 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364947 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364853 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364947 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364891 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364947 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364919 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy\") pod \"1d2cc16e-c9b1-4e60-b632-c824124630c7\" (UID: \"1d2cc16e-c9b1-4e60-b632-c824124630c7\") " Nov 01 07:23:26.364947 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364919 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:26.365233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.364975 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:26.365233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365074 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:23:26.365233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365069 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:26.365233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365200 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:26.365655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365590 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.365655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365621 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.365655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365636 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.365655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365652 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.365655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.365666 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d2cc16e-c9b1-4e60-b632-c824124630c7-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.367004 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.366983 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "kube-api-access-d2hxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:23:26.367004 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.366995 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:23:26.367156 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.367013 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:23:26.367223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.367207 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy" pod "1d2cc16e-c9b1-4e60-b632-c824124630c7" (UID: "1d2cc16e-c9b1-4e60-b632-c824124630c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:23:26.466741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.466707 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2hxh\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-kube-api-access-d2hxh\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.466741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.466740 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.466926 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.466750 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.466926 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.466758 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d2cc16e-c9b1-4e60-b632-c824124630c7-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:23:26.539548 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.539520 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44"] Nov 01 07:23:26.541370 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:23:26.541345 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee388ab_e6b7_4df6_b356_738dd101b16f.slice/crio-35bf483133b508f69bfc73017c712547b0a36de393f887aee0aaabca73895761 WatchSource:0}: Error finding container 35bf483133b508f69bfc73017c712547b0a36de393f887aee0aaabca73895761: Status 404 returned error can't find the container with id 35bf483133b508f69bfc73017c712547b0a36de393f887aee0aaabca73895761 Nov 01 07:23:26.543508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.543476 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:23:26.543569 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.543543 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:23:26.543605 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.543573 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:23:26.842598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.842560 4914 generic.go:356] "Generic (PLEG): container finished" podID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerID="f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb" exitCode=0 Nov 01 07:23:26.843035 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.842624 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" Nov 01 07:23:26.843035 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.842635 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" event={"ID":"1d2cc16e-c9b1-4e60-b632-c824124630c7","Type":"ContainerDied","Data":"f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb"} Nov 01 07:23:26.843035 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.842689 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl" event={"ID":"1d2cc16e-c9b1-4e60-b632-c824124630c7","Type":"ContainerDied","Data":"e9e1d217364def26b79069fab1fe1186344a0c1b3e64c4a35cca33b593054b2e"} Nov 01 07:23:26.843035 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.842712 4914 scope.go:122] "RemoveContainer" containerID="f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb" Nov 01 07:23:26.844464 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.844438 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" event={"ID":"5ee388ab-e6b7-4df6-b356-738dd101b16f","Type":"ContainerStarted","Data":"da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f"} Nov 01 07:23:26.844592 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.844489 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" event={"ID":"5ee388ab-e6b7-4df6-b356-738dd101b16f","Type":"ContainerStarted","Data":"35bf483133b508f69bfc73017c712547b0a36de393f887aee0aaabca73895761"} Nov 01 07:23:26.851698 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.851681 4914 scope.go:122] "RemoveContainer" containerID="06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc" Nov 01 07:23:26.859802 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.859784 4914 scope.go:122] "RemoveContainer" containerID="f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb" Nov 01 07:23:26.860079 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:26.860057 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb\": container with ID starting with f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb not found: ID does not exist" containerID="f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb" Nov 01 07:23:26.860160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.860091 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb"} err="failed to get container status \"f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb\": rpc error: code = NotFound desc = could not find container \"f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb\": container with ID starting with f9b8517132aa10596e8b6a328fbb10e6ce46fd1b127fcd1d752d32dd724285bb not found: ID does not exist" Nov 01 07:23:26.860160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.860114 4914 scope.go:122] "RemoveContainer" containerID="06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc" Nov 01 07:23:26.860438 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:26.860419 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc\": container with ID starting with 06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc not found: ID does not exist" containerID="06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc" Nov 01 07:23:26.860488 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:26.860443 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc"} err="failed to get container status \"06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc\": rpc error: code = NotFound desc = could not find container \"06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc\": container with ID starting with 06db8c64087d68cec94d9320d90ef45bc78081c62ac34bfabf80a275a92a44bc not found: ID does not exist" Nov 01 07:23:27.013223 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.013178 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podStartSLOduration=2.013162412 podStartE2EDuration="2.013162412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:27.010705439 +0000 UTC m=+4422.563494637" watchObservedRunningTime="2026-11-01 07:23:27.013162412 +0000 UTC m=+4422.565951607" Nov 01 07:23:27.070063 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.070038 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:27.075157 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.075134 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:27.132249 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.123061 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl"] Nov 01 07:23:27.137500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.137463 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-976689-openshift-default-5db554f479-mgddl"] Nov 01 07:23:27.852298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.852240 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:27.853308 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:27.853258 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:28.616388 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:28.616352 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" path="/var/lib/kubelet/pods/1d2cc16e-c9b1-4e60-b632-c824124630c7/volumes" Nov 01 07:23:32.596674 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.596641 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc"] Nov 01 07:23:32.597166 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.597071 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" Nov 01 07:23:32.597166 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.597082 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" Nov 01 07:23:32.597247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.597169 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" Nov 01 07:23:32.597247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.597181 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" Nov 01 07:23:32.601645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.601627 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.658174 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.657534 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-501c44d6\"" Nov 01 07:23:32.705380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.705346 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-501c44d6-dockercfg-p5sfj\"" Nov 01 07:23:32.708426 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.708095 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc"] Nov 01 07:23:32.724167 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.724129 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlpk\" (UniqueName: \"kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk\") pod \"payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.724385 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.724337 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.757057 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.757031 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl"] Nov 01 07:23:32.757500 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.757484 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" Nov 01 07:23:32.757571 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.757501 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2cc16e-c9b1-4e60-b632-c824124630c7" containerName="istio-proxy" Nov 01 07:23:32.760718 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.760696 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:32.817450 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.817425 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl"] Nov 01 07:23:32.825199 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.825177 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlpk\" (UniqueName: \"kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk\") pod \"payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.825312 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.825234 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.826070 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.826048 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.881668 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.881610 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlpk\" (UniqueName: \"kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk\") pod \"payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.914797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.914773 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:32.926611 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.926589 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:32.926713 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:32.926641 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx97b\" (UniqueName: \"kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b\") pod \"payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.027912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.027876 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.028073 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.027961 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx97b\" (UniqueName: \"kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b\") pod \"payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.028699 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.028678 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.084383 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.084354 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx97b\" (UniqueName: \"kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b\") pod \"payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.282103 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.282080 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc"] Nov 01 07:23:33.284291 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:23:33.284253 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299cc79f_5eca_4052_9503_17a6bb09cdb7.slice/crio-567e0ebdd80f4f7c970d2b2c381d606c5591bd62f086d1baaea70c17c3a247f9 WatchSource:0}: Error finding container 567e0ebdd80f4f7c970d2b2c381d606c5591bd62f086d1baaea70c17c3a247f9: Status 404 returned error can't find the container with id 567e0ebdd80f4f7c970d2b2c381d606c5591bd62f086d1baaea70c17c3a247f9 Nov 01 07:23:33.373425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.373404 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.739365 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.739337 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl"] Nov 01 07:23:33.740673 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:23:33.740647 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd49266_480c_4244_9294_34ab385844e8.slice/crio-3b8829bd029f2b8acb3b5473a6f8b4b42df97725c975010932f20e35d95ac5fe WatchSource:0}: Error finding container 3b8829bd029f2b8acb3b5473a6f8b4b42df97725c975010932f20e35d95ac5fe: Status 404 returned error can't find the container with id 3b8829bd029f2b8acb3b5473a6f8b4b42df97725c975010932f20e35d95ac5fe Nov 01 07:23:33.884333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.884302 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" event={"ID":"299cc79f-5eca-4052-9503-17a6bb09cdb7","Type":"ContainerStarted","Data":"84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9"} Nov 01 07:23:33.884467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.884353 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" event={"ID":"299cc79f-5eca-4052-9503-17a6bb09cdb7","Type":"ContainerStarted","Data":"567e0ebdd80f4f7c970d2b2c381d606c5591bd62f086d1baaea70c17c3a247f9"} Nov 01 07:23:33.884467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.884398 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:33.886124 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.886089 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" event={"ID":"9bd49266-480c-4244-9294-34ab385844e8","Type":"ContainerStarted","Data":"1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3"} Nov 01 07:23:33.886221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.886141 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" event={"ID":"9bd49266-480c-4244-9294-34ab385844e8","Type":"ContainerStarted","Data":"3b8829bd029f2b8acb3b5473a6f8b4b42df97725c975010932f20e35d95ac5fe"} Nov 01 07:23:33.886221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.886191 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:33.996912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:33.996836 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" podStartSLOduration=1.996819948 podStartE2EDuration="1.996819948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:33.996388066 +0000 UTC m=+4429.549177265" watchObservedRunningTime="2026-11-01 07:23:33.996819948 +0000 UTC m=+4429.549609137" Nov 01 07:23:34.109795 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:34.109733 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" podStartSLOduration=2.109712466 podStartE2EDuration="2.109712466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:34.108717386 +0000 UTC m=+4429.661506580" watchObservedRunningTime="2026-11-01 07:23:34.109712466 +0000 UTC m=+4429.662501662" Nov 01 07:23:35.081895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.081860 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd"] Nov 01 07:23:35.085382 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.085360 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.137141 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.137118 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-501c44d6-serving-cert\"" Nov 01 07:23:35.146492 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.146469 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.146584 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.146509 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.146634 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.146601 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2b8\" (UniqueName: \"kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.146677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.146635 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.188598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.188571 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd"] Nov 01 07:23:35.247605 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.247570 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.247742 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.247634 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2b8\" (UniqueName: \"kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.247742 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.247676 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.247864 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.247785 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.247906 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:35.247875 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-serving-cert: secret "maas-api-e2e-ait-501c44d6-serving-cert" not found Nov 01 07:23:35.247956 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:35.247942 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls podName:2a99d715-db2d-44b9-98ec-d8468ad520ec nodeName:}" failed. No retries permitted until 2026-11-01 07:23:35.747922172 +0000 UTC m=+4431.300711367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls") pod "maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" (UID: "2a99d715-db2d-44b9-98ec-d8468ad520ec") : secret "maas-api-e2e-ait-501c44d6-serving-cert" not found Nov 01 07:23:35.248344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.248326 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.250286 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.250252 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.306550 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.306518 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2b8\" (UniqueName: \"kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.753883 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.753842 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.756415 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.756388 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\") pod \"maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:35.999333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:35.999300 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:36.372729 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:36.372705 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd"] Nov 01 07:23:36.374477 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:23:36.374452 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a99d715_db2d_44b9_98ec_d8468ad520ec.slice/crio-1d054fb5702f6d6a2b2b39ab4756c263ee0423f5a7464eb65bf147f03cd7001e WatchSource:0}: Error finding container 1d054fb5702f6d6a2b2b39ab4756c263ee0423f5a7464eb65bf147f03cd7001e: Status 404 returned error can't find the container with id 1d054fb5702f6d6a2b2b39ab4756c263ee0423f5a7464eb65bf147f03cd7001e Nov 01 07:23:36.375826 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:36.375802 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:23:36.901313 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:36.901258 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" event={"ID":"2a99d715-db2d-44b9-98ec-d8468ad520ec","Type":"ContainerStarted","Data":"1d054fb5702f6d6a2b2b39ab4756c263ee0423f5a7464eb65bf147f03cd7001e"} Nov 01 07:23:37.594378 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:23:37.594262 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:23:37.907869 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:37.907841 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" event={"ID":"2a99d715-db2d-44b9-98ec-d8468ad520ec","Type":"ContainerStarted","Data":"b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340"} Nov 01 07:23:37.908025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:37.907903 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:38.025160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:38.025090 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" podStartSLOduration=2.101598343 podStartE2EDuration="3.025071015s" totalImagesPullingTime="923.472672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:23:38.024089797 +0000 UTC m=+4433.576878992" watchObservedRunningTime="2026-11-01 07:23:38.025071015 +0000 UTC m=+4433.577860210" Nov 01 07:23:43.917757 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:43.917720 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:23:44.894132 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:44.894099 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:23:44.894358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:44.894314 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:23:53.003942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:53.003906 4914 generic.go:356] "Generic (PLEG): container finished" podID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerID="da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f" exitCode=0 Nov 01 07:23:53.004391 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:53.003978 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" event={"ID":"5ee388ab-e6b7-4df6-b356-738dd101b16f","Type":"ContainerDied","Data":"da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f"} Nov 01 07:23:53.004391 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:53.004300 4914 scope.go:122] "RemoveContainer" containerID="da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f" Nov 01 07:23:53.005471 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:53.005438 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:23:53.005544 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:53.005514 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:23:53.005584 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:53.005560 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:23:54.010994 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:54.010963 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" event={"ID":"5ee388ab-e6b7-4df6-b356-738dd101b16f","Type":"ContainerStarted","Data":"eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9"} Nov 01 07:23:54.070493 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:54.070450 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:54.071626 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:54.071593 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 07:23:54.071766 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:54.071648 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 07:23:55.070594 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:55.070556 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 07:23:55.071105 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:55.070604 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 07:23:56.071491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:56.071392 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 07:23:56.071491 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:56.071457 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 07:23:57.070568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:57.070533 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 07:23:57.070815 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:57.070580 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 07:23:57.863301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:57.862016 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:23:58.070675 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:58.070639 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 07:23:58.070876 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:58.070691 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 07:23:59.069811 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:59.069775 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 01 07:23:59.070374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:23:59.069840 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 01 07:24:00.073344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:00.073310 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:24:00.074218 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:00.074199 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:24:07.593902 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:07.593865 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:24:17.052861 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:17.052829 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl"] Nov 01 07:24:17.158399 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:17.158364 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc"] Nov 01 07:24:22.054114 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.054072 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" podUID="9bd49266-480c-4244-9294-34ab385844e8" containerName="payload-processing" containerID="cri-o://1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3" gracePeriod=25 Nov 01 07:24:22.158990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.158959 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" podUID="299cc79f-5eca-4052-9503-17a6bb09cdb7" containerName="payload-pre-processing" containerID="cri-o://84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9" gracePeriod=25 Nov 01 07:24:22.299072 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.299043 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:24:22.398507 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.398484 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:24:22.426242 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.426208 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume\") pod \"9bd49266-480c-4244-9294-34ab385844e8\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " Nov 01 07:24:22.426407 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.426344 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b\" (UniqueName: \"kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b\") pod \"9bd49266-480c-4244-9294-34ab385844e8\" (UID: \"9bd49266-480c-4244-9294-34ab385844e8\") " Nov 01 07:24:22.426585 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.426559 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume" pod "9bd49266-480c-4244-9294-34ab385844e8" (UID: "9bd49266-480c-4244-9294-34ab385844e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:24:22.428481 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.428459 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b" pod "9bd49266-480c-4244-9294-34ab385844e8" (UID: "9bd49266-480c-4244-9294-34ab385844e8"). InnerVolumeSpecName "kube-api-access-kx97b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:24:22.527262 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.527239 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume\") pod \"299cc79f-5eca-4052-9503-17a6bb09cdb7\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " Nov 01 07:24:22.527395 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.527334 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk\" (UniqueName: \"kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk\") pod \"299cc79f-5eca-4052-9503-17a6bb09cdb7\" (UID: \"299cc79f-5eca-4052-9503-17a6bb09cdb7\") " Nov 01 07:24:22.527670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.527626 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx97b\" (UniqueName: \"kubernetes.io/projected/9bd49266-480c-4244-9294-34ab385844e8-kube-api-access-kx97b\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:22.527670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.527636 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume" pod "299cc79f-5eca-4052-9503-17a6bb09cdb7" (UID: "299cc79f-5eca-4052-9503-17a6bb09cdb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:24:22.527670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.527650 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd49266-480c-4244-9294-34ab385844e8-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:22.529757 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.529733 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk" pod "299cc79f-5eca-4052-9503-17a6bb09cdb7" (UID: "299cc79f-5eca-4052-9503-17a6bb09cdb7"). InnerVolumeSpecName "kube-api-access-pqlpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:24:22.628695 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.628674 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299cc79f-5eca-4052-9503-17a6bb09cdb7-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:22.628790 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.628699 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlpk\" (UniqueName: \"kubernetes.io/projected/299cc79f-5eca-4052-9503-17a6bb09cdb7-kube-api-access-pqlpk\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:22.720473 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.720449 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd"] Nov 01 07:24:22.720626 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.720607 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" podUID="2a99d715-db2d-44b9-98ec-d8468ad520ec" containerName="maas-api" containerID="cri-o://b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340" gracePeriod=30 Nov 01 07:24:22.972634 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:22.972611 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:24:23.041946 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.041923 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca\") pod \"2a99d715-db2d-44b9-98ec-d8468ad520ec\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " Nov 01 07:24:23.042101 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.041975 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs\") pod \"2a99d715-db2d-44b9-98ec-d8468ad520ec\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " Nov 01 07:24:23.042101 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.042002 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8\" (UniqueName: \"kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8\") pod \"2a99d715-db2d-44b9-98ec-d8468ad520ec\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " Nov 01 07:24:23.042101 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.042028 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\") pod \"2a99d715-db2d-44b9-98ec-d8468ad520ec\" (UID: \"2a99d715-db2d-44b9-98ec-d8468ad520ec\") " Nov 01 07:24:23.042376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.042354 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca" pod "2a99d715-db2d-44b9-98ec-d8468ad520ec" (UID: "2a99d715-db2d-44b9-98ec-d8468ad520ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:24:23.044120 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.044096 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs" pod "2a99d715-db2d-44b9-98ec-d8468ad520ec" (UID: "2a99d715-db2d-44b9-98ec-d8468ad520ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:24:23.044212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.044170 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls" pod "2a99d715-db2d-44b9-98ec-d8468ad520ec" (UID: "2a99d715-db2d-44b9-98ec-d8468ad520ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:24:23.044263 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.044221 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8" pod "2a99d715-db2d-44b9-98ec-d8468ad520ec" (UID: "2a99d715-db2d-44b9-98ec-d8468ad520ec"). InnerVolumeSpecName "kube-api-access-8p2b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:24:23.143359 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.143239 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a99d715-db2d-44b9-98ec-d8468ad520ec-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:23.143359 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.143270 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:23.143359 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.143303 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2b8\" (UniqueName: \"kubernetes.io/projected/2a99d715-db2d-44b9-98ec-d8468ad520ec-kube-api-access-8p2b8\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:23.143359 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.143318 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a99d715-db2d-44b9-98ec-d8468ad520ec-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:23.147075 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.147046 4914 generic.go:356] "Generic (PLEG): container finished" podID="299cc79f-5eca-4052-9503-17a6bb09cdb7" containerID="84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9" exitCode=0 Nov 01 07:24:23.147207 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.147095 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" Nov 01 07:24:23.147207 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.147137 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" event={"ID":"299cc79f-5eca-4052-9503-17a6bb09cdb7","Type":"ContainerDied","Data":"84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9"} Nov 01 07:24:23.147207 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.147191 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc" event={"ID":"299cc79f-5eca-4052-9503-17a6bb09cdb7","Type":"ContainerDied","Data":"567e0ebdd80f4f7c970d2b2c381d606c5591bd62f086d1baaea70c17c3a247f9"} Nov 01 07:24:23.147364 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.147218 4914 scope.go:122] "RemoveContainer" containerID="84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9" Nov 01 07:24:23.148639 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.148585 4914 generic.go:356] "Generic (PLEG): container finished" podID="2a99d715-db2d-44b9-98ec-d8468ad520ec" containerID="b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340" exitCode=0 Nov 01 07:24:23.148722 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.148633 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" Nov 01 07:24:23.148722 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.148667 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" event={"ID":"2a99d715-db2d-44b9-98ec-d8468ad520ec","Type":"ContainerDied","Data":"b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340"} Nov 01 07:24:23.148841 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.148739 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd" event={"ID":"2a99d715-db2d-44b9-98ec-d8468ad520ec","Type":"ContainerDied","Data":"1d054fb5702f6d6a2b2b39ab4756c263ee0423f5a7464eb65bf147f03cd7001e"} Nov 01 07:24:23.150340 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.150315 4914 generic.go:356] "Generic (PLEG): container finished" podID="9bd49266-480c-4244-9294-34ab385844e8" containerID="1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3" exitCode=0 Nov 01 07:24:23.150503 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.150390 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" event={"ID":"9bd49266-480c-4244-9294-34ab385844e8","Type":"ContainerDied","Data":"1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3"} Nov 01 07:24:23.150503 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.150434 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" event={"ID":"9bd49266-480c-4244-9294-34ab385844e8","Type":"ContainerDied","Data":"3b8829bd029f2b8acb3b5473a6f8b4b42df97725c975010932f20e35d95ac5fe"} Nov 01 07:24:23.150503 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.150450 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl" Nov 01 07:24:23.161579 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.161559 4914 scope.go:122] "RemoveContainer" containerID="84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9" Nov 01 07:24:23.161867 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:23.161837 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9\": container with ID starting with 84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9 not found: ID does not exist" containerID="84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9" Nov 01 07:24:23.161925 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.161865 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9"} err="failed to get container status \"84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9\": rpc error: code = NotFound desc = could not find container \"84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9\": container with ID starting with 84606b5260447e02e85756b8d580d418857f7534ef650fe2e62a22e3f91cd8c9 not found: ID does not exist" Nov 01 07:24:23.161925 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.161889 4914 scope.go:122] "RemoveContainer" containerID="b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340" Nov 01 07:24:23.170233 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.170217 4914 scope.go:122] "RemoveContainer" containerID="b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340" Nov 01 07:24:23.170521 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:23.170502 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340\": container with ID starting with b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340 not found: ID does not exist" containerID="b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340" Nov 01 07:24:23.170594 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.170526 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340"} err="failed to get container status \"b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340\": rpc error: code = NotFound desc = could not find container \"b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340\": container with ID starting with b05df05bd4d59cec9126644340b080e964afbdf04bcdf14640065cb073fcb340 not found: ID does not exist" Nov 01 07:24:23.170594 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.170542 4914 scope.go:122] "RemoveContainer" containerID="1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3" Nov 01 07:24:23.177847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.177830 4914 scope.go:122] "RemoveContainer" containerID="1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3" Nov 01 07:24:23.178099 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:23.178079 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3\": container with ID starting with 1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3 not found: ID does not exist" containerID="1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3" Nov 01 07:24:23.178162 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.178109 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3"} err="failed to get container status \"1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3\": rpc error: code = NotFound desc = could not find container \"1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3\": container with ID starting with 1062c79c05782b4bac73373e98a5f8b6440e1b7710217ec8762b6bf3e5dd7ae3 not found: ID does not exist" Nov 01 07:24:23.313963 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.313921 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc"] Nov 01 07:24:23.314159 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.313986 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-501c44d6-58df7fb659-jxhwc"] Nov 01 07:24:23.418966 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.418893 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl"] Nov 01 07:24:23.423247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.423222 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-501c44d6-5d8fc4b77-sf2gl"] Nov 01 07:24:23.531081 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.531053 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd"] Nov 01 07:24:23.533875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:23.533849 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-501c44d6-7c79c9f9cf-t8rdd"] Nov 01 07:24:24.628038 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:24.627764 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299cc79f-5eca-4052-9503-17a6bb09cdb7" path="/var/lib/kubelet/pods/299cc79f-5eca-4052-9503-17a6bb09cdb7/volumes" Nov 01 07:24:24.628599 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:24.628403 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a99d715-db2d-44b9-98ec-d8468ad520ec" path="/var/lib/kubelet/pods/2a99d715-db2d-44b9-98ec-d8468ad520ec/volumes" Nov 01 07:24:24.628929 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:24.628897 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd49266-480c-4244-9294-34ab385844e8" path="/var/lib/kubelet/pods/9bd49266-480c-4244-9294-34ab385844e8/volumes" Nov 01 07:24:26.683977 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:26.683947 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44"] Nov 01 07:24:26.684437 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:26.684143 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" containerID="cri-o://eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9" gracePeriod=30 Nov 01 07:24:30.076530 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:30.076493 4914 patch_prober.go:29] interesting pod/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 01 07:24:30.077010 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:30.076552 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 07:24:31.941913 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:31.941889 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:24:32.126481 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126456 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126491 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126535 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126583 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126635 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126682 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126734 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126721 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:24:32.126882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126777 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.126882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126778 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:24:32.126882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126826 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token\") pod \"5ee388ab-e6b7-4df6-b356-738dd101b16f\" (UID: \"5ee388ab-e6b7-4df6-b356-738dd101b16f\") " Nov 01 07:24:32.127187 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.126996 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:24:32.127187 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.127092 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:24:32.127321 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.127195 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:24:32.127321 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.127288 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee388ab-e6b7-4df6-b356-738dd101b16f-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.127321 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.127310 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.127643 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.127324 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.127643 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.127338 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.129027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.128994 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:24:32.129146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.129045 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:24:32.129146 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.129120 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:24:32.129346 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.129324 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5" pod "5ee388ab-e6b7-4df6-b356-738dd101b16f" (UID: "5ee388ab-e6b7-4df6-b356-738dd101b16f"). InnerVolumeSpecName "kube-api-access-qxhj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:24:32.199501 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.199475 4914 generic.go:356] "Generic (PLEG): container finished" podID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerID="eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9" exitCode=0 Nov 01 07:24:32.199615 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.199521 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" Nov 01 07:24:32.199615 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.199557 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" event={"ID":"5ee388ab-e6b7-4df6-b356-738dd101b16f","Type":"ContainerDied","Data":"eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9"} Nov 01 07:24:32.199615 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.199606 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44" event={"ID":"5ee388ab-e6b7-4df6-b356-738dd101b16f","Type":"ContainerDied","Data":"35bf483133b508f69bfc73017c712547b0a36de393f887aee0aaabca73895761"} Nov 01 07:24:32.199757 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.199628 4914 scope.go:122] "RemoveContainer" containerID="eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9" Nov 01 07:24:32.208476 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.208460 4914 scope.go:122] "RemoveContainer" containerID="da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f" Nov 01 07:24:32.216826 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.216802 4914 scope.go:122] "RemoveContainer" containerID="eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9" Nov 01 07:24:32.217081 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:32.217065 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9\": container with ID starting with eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9 not found: ID does not exist" containerID="eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9" Nov 01 07:24:32.217129 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.217090 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9"} err="failed to get container status \"eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9\": rpc error: code = NotFound desc = could not find container \"eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9\": container with ID starting with eda5d4f7170a503825ec5ca349b1e91d644da06bb92af51de62b070968b69cc9 not found: ID does not exist" Nov 01 07:24:32.217129 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.217107 4914 scope.go:122] "RemoveContainer" containerID="da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f" Nov 01 07:24:32.217353 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:32.217337 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f\": container with ID starting with da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f not found: ID does not exist" containerID="da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f" Nov 01 07:24:32.217415 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.217357 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f"} err="failed to get container status \"da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f\": rpc error: code = NotFound desc = could not find container \"da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f\": container with ID starting with da0c6273eeedc33cf2b2b90a1e6712c9b1be6eeac0b6c3b03f77424579573f5f not found: ID does not exist" Nov 01 07:24:32.227794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.227777 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.227864 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.227795 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.227864 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.227805 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.227864 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.227814 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhj5\" (UniqueName: \"kubernetes.io/projected/5ee388ab-e6b7-4df6-b356-738dd101b16f-kube-api-access-qxhj5\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.227864 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.227824 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee388ab-e6b7-4df6-b356-738dd101b16f-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:24:32.361719 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.361688 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44"] Nov 01 07:24:32.368015 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.367996 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-501c44d6-openshift-default-74b687d8f5-pvq44"] Nov 01 07:24:32.616164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:32.616139 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" path="/var/lib/kubelet/pods/5ee388ab-e6b7-4df6-b356-738dd101b16f/volumes" Nov 01 07:24:37.592349 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:24:37.592313 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:24:45.174429 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:45.174404 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:24:45.196708 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:45.196689 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:24:58.820305 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:58.817378 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:24:58.820305 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:24:58.817496 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:25:07.677162 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:25:07.677106 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:25:37.592253 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:25:37.592161 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:25:57.713859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.713821 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md"] Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714246 4914 cpu_manager.go:414] "containerMap: removing container" podUID="299cc79f-5eca-4052-9503-17a6bb09cdb7" containerName="payload-pre-processing" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714258 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299cc79f-5eca-4052-9503-17a6bb09cdb7" containerName="payload-pre-processing" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714303 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714310 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714323 4914 cpu_manager.go:414] "containerMap: removing container" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714329 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714338 4914 cpu_manager.go:414] "containerMap: removing container" podUID="2a99d715-db2d-44b9-98ec-d8468ad520ec" containerName="maas-api" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714343 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a99d715-db2d-44b9-98ec-d8468ad520ec" containerName="maas-api" Nov 01 07:25:57.714358 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714358 4914 cpu_manager.go:414] "containerMap: removing container" podUID="9bd49266-480c-4244-9294-34ab385844e8" containerName="payload-processing" Nov 01 07:25:57.714689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714366 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd49266-480c-4244-9294-34ab385844e8" containerName="payload-processing" Nov 01 07:25:57.714689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714435 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd49266-480c-4244-9294-34ab385844e8" containerName="payload-processing" Nov 01 07:25:57.714689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714450 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="299cc79f-5eca-4052-9503-17a6bb09cdb7" containerName="payload-pre-processing" Nov 01 07:25:57.714689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714462 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" Nov 01 07:25:57.714689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714470 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee388ab-e6b7-4df6-b356-738dd101b16f" containerName="istio-proxy" Nov 01 07:25:57.714689 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.714477 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a99d715-db2d-44b9-98ec-d8468ad520ec" containerName="maas-api" Nov 01 07:25:57.717551 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.717532 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:57.807962 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.807931 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7a816b61-kserve-self-signed-certs\"" Nov 01 07:25:57.809846 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.809820 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md"] Nov 01 07:25:57.821311 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.821257 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7a816b61-epp-sa-dockercfg-cttnk\"" Nov 01 07:25:57.898208 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.898161 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs\") pod \"e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:57.898523 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.898486 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk5d8\" (UniqueName: \"kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8\") pod \"e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:57.999942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.999871 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk5d8\" (UniqueName: \"kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8\") pod \"e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:57.999942 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:57.999930 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs\") pod \"e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:58.002532 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:58.002510 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs\") pod \"e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:58.055653 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:58.055634 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk5d8\" (UniqueName: \"kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8\") pod \"e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:58.330660 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:58.330581 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:25:58.698376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:58.698351 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md"] Nov 01 07:25:58.700547 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:25:58.700518 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeba6bc_da14_4362_8840_b2bd990694b9.slice/crio-e5e25fb92aef56af4e415765be302b52619ce7bce118464b4a8acb9b147df097 WatchSource:0}: Error finding container e5e25fb92aef56af4e415765be302b52619ce7bce118464b4a8acb9b147df097: Status 404 returned error can't find the container with id e5e25fb92aef56af4e415765be302b52619ce7bce118464b4a8acb9b147df097 Nov 01 07:25:58.787944 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:25:58.787918 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" event={"ID":"ddeba6bc-da14-4362-8840-b2bd990694b9","Type":"ContainerStarted","Data":"e5e25fb92aef56af4e415765be302b52619ce7bce118464b4a8acb9b147df097"} Nov 01 07:26:03.052515 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:26:03.052462 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:26:03.053665 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:26:03.053235 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:26:03.883311 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:26:03.882694 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" event={"ID":"ddeba6bc-da14-4362-8840-b2bd990694b9","Type":"ContainerStarted","Data":"a07d19c79075eb031fb97c5fcc649a5a2129f8f6be0fb006b8c2dd05c86d75bc"} Nov 01 07:26:03.887310 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:26:03.883594 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:26:03.998351 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:26:03.997761 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" podStartSLOduration=2.053086338 podStartE2EDuration="6.997741009s" totalImagesPullingTime="4.944654671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:26:03.995785589 +0000 UTC m=+4579.548574774" watchObservedRunningTime="2026-11-01 07:26:03.997741009 +0000 UTC m=+4579.550530205" Nov 01 07:26:07.591164 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:26:07.591127 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:26:35.894895 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:26:35.894864 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:26:37.592040 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:26:37.592005 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:27:07.591737 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:27:07.591654 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:27:10.348846 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:10.348805 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md"] Nov 01 07:27:15.906040 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:15.905988 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:27:18.067299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:18.066368 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:27:18.067299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:18.066509 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:27:25.369541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:25.369478 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" containerID="cri-o://a07d19c79075eb031fb97c5fcc649a5a2129f8f6be0fb006b8c2dd05c86d75bc" gracePeriod=45 Nov 01 07:27:25.895085 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:25.895044 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:27:31.759182 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:31.759143 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:27:31.759912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:31.759840 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:27:35.894302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:35.894250 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:27:35.894712 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:35.894397 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:27:37.591719 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:27:37.591676 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:27:45.894541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:45.894497 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:27:55.490527 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.490331 4914 generic.go:356] "Generic (PLEG): container finished" podID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerID="a07d19c79075eb031fb97c5fcc649a5a2129f8f6be0fb006b8c2dd05c86d75bc" exitCode=0 Nov 01 07:27:55.490527 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.490405 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" event={"ID":"ddeba6bc-da14-4362-8840-b2bd990694b9","Type":"ContainerDied","Data":"a07d19c79075eb031fb97c5fcc649a5a2129f8f6be0fb006b8c2dd05c86d75bc"} Nov 01 07:27:55.626350 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.626320 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:27:55.681793 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.681730 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8\" (UniqueName: \"kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8\") pod \"ddeba6bc-da14-4362-8840-b2bd990694b9\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " Nov 01 07:27:55.681793 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.681765 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs\") pod \"ddeba6bc-da14-4362-8840-b2bd990694b9\" (UID: \"ddeba6bc-da14-4362-8840-b2bd990694b9\") " Nov 01 07:27:55.684066 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.684044 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs" pod "ddeba6bc-da14-4362-8840-b2bd990694b9" (UID: "ddeba6bc-da14-4362-8840-b2bd990694b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:27:55.684159 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.684065 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8" pod "ddeba6bc-da14-4362-8840-b2bd990694b9" (UID: "ddeba6bc-da14-4362-8840-b2bd990694b9"). InnerVolumeSpecName "kube-api-access-tk5d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:27:55.783133 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.783104 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk5d8\" (UniqueName: \"kubernetes.io/projected/ddeba6bc-da14-4362-8840-b2bd990694b9-kube-api-access-tk5d8\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:55.783133 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:55.783130 4914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddeba6bc-da14-4362-8840-b2bd990694b9-tls-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:27:56.496135 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:56.496112 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" Nov 01 07:27:56.496583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:56.496154 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md" event={"ID":"ddeba6bc-da14-4362-8840-b2bd990694b9","Type":"ContainerDied","Data":"e5e25fb92aef56af4e415765be302b52619ce7bce118464b4a8acb9b147df097"} Nov 01 07:27:56.496583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:56.496213 4914 scope.go:122] "RemoveContainer" containerID="a07d19c79075eb031fb97c5fcc649a5a2129f8f6be0fb006b8c2dd05c86d75bc" Nov 01 07:27:56.662952 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:56.662925 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md"] Nov 01 07:27:56.709924 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:56.709899 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-7a816b61-kserve-router-scheduler-cccd88cf6-4k8md"] Nov 01 07:27:58.614825 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:27:58.614795 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" path="/var/lib/kubelet/pods/ddeba6bc-da14-4362-8840-b2bd990694b9/volumes" Nov 01 07:28:03.549577 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.549544 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf"] Nov 01 07:28:03.550035 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.549962 4914 cpu_manager.go:414] "containerMap: removing container" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" Nov 01 07:28:03.550035 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.549989 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" Nov 01 07:28:03.550107 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.550074 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeba6bc-da14-4362-8840-b2bd990694b9" containerName="main" Nov 01 07:28:03.554567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.554547 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.640540 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.640504 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.640707 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.640600 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.640707 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.640690 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.640802 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.640715 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.640802 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.640786 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.640921 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.640904 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497q9\" (UniqueName: \"kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.657155 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.657135 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fff8f9af-kserve-self-signed-certs\"" Nov 01 07:28:03.658922 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.658903 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf"] Nov 01 07:28:03.741539 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.741504 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497q9\" (UniqueName: \"kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.741732 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.741587 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.741732 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.741640 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.741732 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.741685 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.741732 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.741708 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.741975 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.741735 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.742095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.742059 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.742220 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.742103 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.742220 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.742139 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.744099 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.744075 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.744367 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.744346 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.797868 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.797842 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-497q9\" (UniqueName: \"kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9\") pod \"e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:03.866109 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:03.866091 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:04.245780 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:04.245756 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf"] Nov 01 07:28:04.247677 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:28:04.247648 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d2aadb1_ca6e_46e3_9026_18c443bc8a61.slice/crio-f7eb92e1d088ba7d46e976d9f0e4764f7ecdfb886c9931ff77e8c22b86bb42cf WatchSource:0}: Error finding container f7eb92e1d088ba7d46e976d9f0e4764f7ecdfb886c9931ff77e8c22b86bb42cf: Status 404 returned error can't find the container with id f7eb92e1d088ba7d46e976d9f0e4764f7ecdfb886c9931ff77e8c22b86bb42cf Nov 01 07:28:04.546312 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:04.546211 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" event={"ID":"9d2aadb1-ca6e-46e3-9026-18c443bc8a61","Type":"ContainerStarted","Data":"5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e"} Nov 01 07:28:04.546312 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:04.546254 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" event={"ID":"9d2aadb1-ca6e-46e3-9026-18c443bc8a61","Type":"ContainerStarted","Data":"f7eb92e1d088ba7d46e976d9f0e4764f7ecdfb886c9931ff77e8c22b86bb42cf"} Nov 01 07:28:04.659268 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:04.659215 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" podStartSLOduration=1.659197867 podStartE2EDuration="1.659197867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:28:04.657797608 +0000 UTC m=+4700.210586802" watchObservedRunningTime="2026-11-01 07:28:04.659197867 +0000 UTC m=+4700.211987063" Nov 01 07:28:04.866670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:04.866636 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:07.590682 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:28:07.590648 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:28:13.866482 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:13.866451 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:13.878349 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:13.878329 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:14.641010 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:14.640986 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:28:37.591744 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:28:37.591657 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:28:58.816299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:58.815399 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:28:58.816299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:28:58.815473 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:29:03.559378 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:03.559347 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf"] Nov 01 07:29:07.591131 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:29:07.591100 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:29:18.578998 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.578943 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" podUID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" containerName="main" containerID="cri-o://5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e" gracePeriod=45 Nov 01 07:29:18.631100 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.631069 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" podUID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" containerName="main" probeResult="failure" output="Get \"https://10.132.0.185:8000/ready\": dial tcp 10.132.0.185:8000: connect: connection refused" Nov 01 07:29:18.836912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.836852 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:29:18.968880 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.968850 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir\") pod \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " Nov 01 07:29:18.969077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.968920 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9\" (UniqueName: \"kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9\") pod \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " Nov 01 07:29:18.969077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.968974 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache\") pod \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " Nov 01 07:29:18.969077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969015 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home\") pod \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " Nov 01 07:29:18.969077 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969054 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs\") pod \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " Nov 01 07:29:18.969333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969143 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir" pod "9d2aadb1-ca6e-46e3-9026-18c443bc8a61" (UID: "9d2aadb1-ca6e-46e3-9026-18c443bc8a61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:18.969333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969185 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm\") pod \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\" (UID: \"9d2aadb1-ca6e-46e3-9026-18c443bc8a61\") " Nov 01 07:29:18.969333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969261 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache" pod "9d2aadb1-ca6e-46e3-9026-18c443bc8a61" (UID: "9d2aadb1-ca6e-46e3-9026-18c443bc8a61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:18.969474 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969391 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home" pod "9d2aadb1-ca6e-46e3-9026-18c443bc8a61" (UID: "9d2aadb1-ca6e-46e3-9026-18c443bc8a61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:18.969576 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969556 4914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tmp-dir\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:18.969633 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969582 4914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-model-cache\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:18.969633 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.969596 4914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-home\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:18.971361 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.971333 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs" pod "9d2aadb1-ca6e-46e3-9026-18c443bc8a61" (UID: "9d2aadb1-ca6e-46e3-9026-18c443bc8a61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:29:18.971494 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.971415 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9" pod "9d2aadb1-ca6e-46e3-9026-18c443bc8a61" (UID: "9d2aadb1-ca6e-46e3-9026-18c443bc8a61"). InnerVolumeSpecName "kube-api-access-497q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:29:18.971755 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:18.971732 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm" pod "9d2aadb1-ca6e-46e3-9026-18c443bc8a61" (UID: "9d2aadb1-ca6e-46e3-9026-18c443bc8a61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:29:19.070570 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.070548 4914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-tls-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:19.070570 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.070568 4914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-dshm\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:19.070713 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.070577 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497q9\" (UniqueName: \"kubernetes.io/projected/9d2aadb1-ca6e-46e3-9026-18c443bc8a61-kube-api-access-497q9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:29:19.087167 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.087108 4914 generic.go:356] "Generic (PLEG): container finished" podID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" containerID="5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e" exitCode=0 Nov 01 07:29:19.087167 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.087152 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" Nov 01 07:29:19.087302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.087162 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" event={"ID":"9d2aadb1-ca6e-46e3-9026-18c443bc8a61","Type":"ContainerDied","Data":"5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e"} Nov 01 07:29:19.087302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.087208 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf" event={"ID":"9d2aadb1-ca6e-46e3-9026-18c443bc8a61","Type":"ContainerDied","Data":"f7eb92e1d088ba7d46e976d9f0e4764f7ecdfb886c9931ff77e8c22b86bb42cf"} Nov 01 07:29:19.087302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.087226 4914 scope.go:122] "RemoveContainer" containerID="5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e" Nov 01 07:29:19.096230 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.096216 4914 scope.go:122] "RemoveContainer" containerID="5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e" Nov 01 07:29:19.096520 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:29:19.096503 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e\": container with ID starting with 5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e not found: ID does not exist" containerID="5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e" Nov 01 07:29:19.096589 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.096527 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e"} err="failed to get container status \"5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e\": rpc error: code = NotFound desc = could not find container \"5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e\": container with ID starting with 5f732e21c0615af7dc05e954e19e3fe9d6c476e74d0d6c7316c6ea4b9798af6e not found: ID does not exist" Nov 01 07:29:19.248131 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.248110 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf"] Nov 01 07:29:19.249710 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:19.249692 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fff8f9af-kserve-6cf9476949-rh2hf"] Nov 01 07:29:20.615261 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:20.615236 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" path="/var/lib/kubelet/pods/9d2aadb1-ca6e-46e3-9026-18c443bc8a61/volumes" Nov 01 07:29:28.812639 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:28.812596 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-qpmzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:29:28.813346 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:28.813309 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-qpmzx" podUID="f393dcd3-3bd8-430f-aa5c-67af99441bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:29:37.591864 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:29:37.591826 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:29:45.201499 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.201473 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:29:45.224803 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.224773 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:29:45.783155 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.783124 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-56f8865864-tcnsn"] Nov 01 07:29:45.783538 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.783526 4914 cpu_manager.go:414] "containerMap: removing container" podUID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" containerName="main" Nov 01 07:29:45.783592 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.783540 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" containerName="main" Nov 01 07:29:45.783642 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.783633 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2aadb1-ca6e-46e3-9026-18c443bc8a61" containerName="main" Nov 01 07:29:45.786667 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.786650 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:45.843109 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.843087 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-56f8865864-tcnsn"] Nov 01 07:29:45.968602 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.968563 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume\") pod \"payload-pre-processing-56f8865864-tcnsn\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:45.968839 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:45.968775 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtw9\" (UniqueName: \"kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9\") pod \"payload-pre-processing-56f8865864-tcnsn\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:46.069820 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:46.069728 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume\") pod \"payload-pre-processing-56f8865864-tcnsn\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:46.070010 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:46.069833 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtw9\" (UniqueName: \"kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9\") pod \"payload-pre-processing-56f8865864-tcnsn\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:46.070560 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:46.070536 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume\") pod \"payload-pre-processing-56f8865864-tcnsn\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:46.126617 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:46.126588 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtw9\" (UniqueName: \"kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9\") pod \"payload-pre-processing-56f8865864-tcnsn\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:46.398472 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:46.398450 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:46.767306 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:46.767221 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-56f8865864-tcnsn"] Nov 01 07:29:46.773357 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:29:46.773334 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd946dd9f_2e33_4fea_9607_1d7be53372aa.slice/crio-2889392875ca71babf165939a4032b223ab445fd546eb5b3ddc362112c2102b2 WatchSource:0}: Error finding container 2889392875ca71babf165939a4032b223ab445fd546eb5b3ddc362112c2102b2: Status 404 returned error can't find the container with id 2889392875ca71babf165939a4032b223ab445fd546eb5b3ddc362112c2102b2 Nov 01 07:29:47.280368 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:47.280338 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" event={"ID":"d946dd9f-2e33-4fea-9607-1d7be53372aa","Type":"ContainerStarted","Data":"8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a"} Nov 01 07:29:47.280513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:47.280383 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" event={"ID":"d946dd9f-2e33-4fea-9607-1d7be53372aa","Type":"ContainerStarted","Data":"2889392875ca71babf165939a4032b223ab445fd546eb5b3ddc362112c2102b2"} Nov 01 07:29:47.280513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:47.280457 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:47.393234 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:47.393183 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" podStartSLOduration=2.393171657 podStartE2EDuration="2.393171657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:29:47.391226493 +0000 UTC m=+4802.944015688" watchObservedRunningTime="2026-11-01 07:29:47.393171657 +0000 UTC m=+4802.945960845" Nov 01 07:29:58.287883 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:58.287807 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:29:58.451251 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:29:58.451223 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7dngs"] Nov 01 07:30:00.417443 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.417413 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-54ddd7df84-r5p2d"] Nov 01 07:30:00.420841 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.420826 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.477270 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.477245 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-54ddd7df84-r5p2d"] Nov 01 07:30:00.492464 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.492443 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgkj\" (UniqueName: \"kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj\") pod \"payload-processing-54ddd7df84-r5p2d\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.492560 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.492479 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume\") pod \"payload-processing-54ddd7df84-r5p2d\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.593587 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.593560 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume\") pod \"payload-processing-54ddd7df84-r5p2d\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.593705 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.593660 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgkj\" (UniqueName: \"kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj\") pod \"payload-processing-54ddd7df84-r5p2d\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.594219 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.594201 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume\") pod \"payload-processing-54ddd7df84-r5p2d\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.649930 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.649901 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgkj\" (UniqueName: \"kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj\") pod \"payload-processing-54ddd7df84-r5p2d\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:00.733501 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:00.733405 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:01.106937 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:01.106912 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-54ddd7df84-r5p2d"] Nov 01 07:30:01.108543 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:30:01.108512 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb949b69f_92fc_4690_9f87_e3b54f6e515a.slice/crio-18595163f1d9a393a50aadef985aa082fabcf3dcaf5d0e655e1f6faf020cd721 WatchSource:0}: Error finding container 18595163f1d9a393a50aadef985aa082fabcf3dcaf5d0e655e1f6faf020cd721: Status 404 returned error can't find the container with id 18595163f1d9a393a50aadef985aa082fabcf3dcaf5d0e655e1f6faf020cd721 Nov 01 07:30:01.345078 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:01.345034 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" event={"ID":"b949b69f-92fc-4690-9f87-e3b54f6e515a","Type":"ContainerStarted","Data":"b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180"} Nov 01 07:30:01.345247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:01.345096 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" event={"ID":"b949b69f-92fc-4690-9f87-e3b54f6e515a","Type":"ContainerStarted","Data":"18595163f1d9a393a50aadef985aa082fabcf3dcaf5d0e655e1f6faf020cd721"} Nov 01 07:30:01.345247 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:01.345118 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:01.457484 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:01.457376 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" podStartSLOduration=1.457361777 podStartE2EDuration="1.457361777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:30:01.456723006 +0000 UTC m=+4817.009512199" watchObservedRunningTime="2026-11-01 07:30:01.457361777 +0000 UTC m=+4817.010150971" Nov 01 07:30:03.452158 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.452123 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" podUID="91e538ea-0732-44fd-bf96-f0e0131e0cdf" containerName="payload-pre-processing" containerID="cri-o://77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c" gracePeriod=25 Nov 01 07:30:03.697875 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.697850 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 07:30:03.720598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.720532 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp\" (UniqueName: \"kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp\") pod \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " Nov 01 07:30:03.720598 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.720573 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume\") pod \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\" (UID: \"91e538ea-0732-44fd-bf96-f0e0131e0cdf\") " Nov 01 07:30:03.720990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.720967 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume" pod "91e538ea-0732-44fd-bf96-f0e0131e0cdf" (UID: "91e538ea-0732-44fd-bf96-f0e0131e0cdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:30:03.722853 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.722831 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp" pod "91e538ea-0732-44fd-bf96-f0e0131e0cdf" (UID: "91e538ea-0732-44fd-bf96-f0e0131e0cdf"). InnerVolumeSpecName "kube-api-access-ndnqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:30:03.821861 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.821836 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnqp\" (UniqueName: \"kubernetes.io/projected/91e538ea-0732-44fd-bf96-f0e0131e0cdf-kube-api-access-ndnqp\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:03.821861 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:03.821859 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e538ea-0732-44fd-bf96-f0e0131e0cdf-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:04.360519 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.360483 4914 generic.go:356] "Generic (PLEG): container finished" podID="91e538ea-0732-44fd-bf96-f0e0131e0cdf" containerID="77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c" exitCode=0 Nov 01 07:30:04.360691 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.360560 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" Nov 01 07:30:04.360691 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.360558 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" event={"ID":"91e538ea-0732-44fd-bf96-f0e0131e0cdf","Type":"ContainerDied","Data":"77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c"} Nov 01 07:30:04.360763 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.360695 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7dngs" event={"ID":"91e538ea-0732-44fd-bf96-f0e0131e0cdf","Type":"ContainerDied","Data":"3e6d83225c379f187cf9d4419472fa33323e62ee748b3fd5f0f768766dcb5935"} Nov 01 07:30:04.360763 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.360726 4914 scope.go:122] "RemoveContainer" containerID="77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c" Nov 01 07:30:04.370227 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.370209 4914 scope.go:122] "RemoveContainer" containerID="77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c" Nov 01 07:30:04.370544 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:30:04.370517 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c\": container with ID starting with 77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c not found: ID does not exist" containerID="77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c" Nov 01 07:30:04.370645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.370551 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c"} err="failed to get container status \"77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c\": rpc error: code = NotFound desc = could not find container \"77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c\": container with ID starting with 77ff99f43f0c75c84712671b5b527bd1bfb9c28e30ca1905ad5350a8078a7c2c not found: ID does not exist" Nov 01 07:30:04.521340 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.521311 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7dngs"] Nov 01 07:30:04.570620 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.570603 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7dngs"] Nov 01 07:30:04.614907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:04.614860 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e538ea-0732-44fd-bf96-f0e0131e0cdf" path="/var/lib/kubelet/pods/91e538ea-0732-44fd-bf96-f0e0131e0cdf/volumes" Nov 01 07:30:07.590860 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:30:07.590827 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:30:12.352440 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:12.352416 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:12.516136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:12.516104 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-q2fb6"] Nov 01 07:30:14.681920 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.681891 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6759564b6c-wpg92"] Nov 01 07:30:14.682324 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.682310 4914 cpu_manager.go:414] "containerMap: removing container" podUID="91e538ea-0732-44fd-bf96-f0e0131e0cdf" containerName="payload-pre-processing" Nov 01 07:30:14.682376 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.682325 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e538ea-0732-44fd-bf96-f0e0131e0cdf" containerName="payload-pre-processing" Nov 01 07:30:14.682412 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.682384 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e538ea-0732-44fd-bf96-f0e0131e0cdf" containerName="payload-pre-processing" Nov 01 07:30:14.686566 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.686548 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.742747 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.742723 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6759564b6c-wpg92"] Nov 01 07:30:14.812740 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.812716 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5t7x\" (UniqueName: \"kubernetes.io/projected/402eb62b-368e-447c-be22-a1e68493ad7c-kube-api-access-f5t7x\") pod \"payload-processing-6759564b6c-wpg92\" (UID: \"402eb62b-368e-447c-be22-a1e68493ad7c\") " pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.812860 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.812756 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402eb62b-368e-447c-be22-a1e68493ad7c-plugins-config-volume\") pod \"payload-processing-6759564b6c-wpg92\" (UID: \"402eb62b-368e-447c-be22-a1e68493ad7c\") " pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.849152 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.849126 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z"] Nov 01 07:30:14.852961 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.852945 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:14.908648 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.908626 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z"] Nov 01 07:30:14.914377 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.914347 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4gw\" (UniqueName: \"kubernetes.io/projected/ec5403c5-bc63-43e7-8702-eebf3ffd6cd9-kube-api-access-cc4gw\") pod \"payload-pre-processing-5b4cf47bfb-xjj8z\" (UID: \"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9\") " pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:14.914495 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.914402 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5t7x\" (UniqueName: \"kubernetes.io/projected/402eb62b-368e-447c-be22-a1e68493ad7c-kube-api-access-f5t7x\") pod \"payload-processing-6759564b6c-wpg92\" (UID: \"402eb62b-368e-447c-be22-a1e68493ad7c\") " pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.914567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.914501 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402eb62b-368e-447c-be22-a1e68493ad7c-plugins-config-volume\") pod \"payload-processing-6759564b6c-wpg92\" (UID: \"402eb62b-368e-447c-be22-a1e68493ad7c\") " pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.914626 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.914562 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5403c5-bc63-43e7-8702-eebf3ffd6cd9-plugins-config-volume\") pod \"payload-pre-processing-5b4cf47bfb-xjj8z\" (UID: \"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9\") " pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:14.915126 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.915107 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402eb62b-368e-447c-be22-a1e68493ad7c-plugins-config-volume\") pod \"payload-processing-6759564b6c-wpg92\" (UID: \"402eb62b-368e-447c-be22-a1e68493ad7c\") " pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.974647 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.974593 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5t7x\" (UniqueName: \"kubernetes.io/projected/402eb62b-368e-447c-be22-a1e68493ad7c-kube-api-access-f5t7x\") pod \"payload-processing-6759564b6c-wpg92\" (UID: \"402eb62b-368e-447c-be22-a1e68493ad7c\") " pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:14.999115 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:14.999093 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:15.015296 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.015258 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5403c5-bc63-43e7-8702-eebf3ffd6cd9-plugins-config-volume\") pod \"payload-pre-processing-5b4cf47bfb-xjj8z\" (UID: \"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9\") " pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:15.015395 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.015353 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4gw\" (UniqueName: \"kubernetes.io/projected/ec5403c5-bc63-43e7-8702-eebf3ffd6cd9-kube-api-access-cc4gw\") pod \"payload-pre-processing-5b4cf47bfb-xjj8z\" (UID: \"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9\") " pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:15.016061 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.016039 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5403c5-bc63-43e7-8702-eebf3ffd6cd9-plugins-config-volume\") pod \"payload-pre-processing-5b4cf47bfb-xjj8z\" (UID: \"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9\") " pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:15.071511 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.071482 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4gw\" (UniqueName: \"kubernetes.io/projected/ec5403c5-bc63-43e7-8702-eebf3ffd6cd9-kube-api-access-cc4gw\") pod \"payload-pre-processing-5b4cf47bfb-xjj8z\" (UID: \"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9\") " pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:15.165516 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.165484 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:15.365741 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.365712 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6759564b6c-wpg92"] Nov 01 07:30:15.367085 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:30:15.367061 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402eb62b_368e_447c_be22_a1e68493ad7c.slice/crio-21ed1a8e26da25c2495de3e32daf17ddcff9c4699ca8bfa6b5d01da2d9997a9f WatchSource:0}: Error finding container 21ed1a8e26da25c2495de3e32daf17ddcff9c4699ca8bfa6b5d01da2d9997a9f: Status 404 returned error can't find the container with id 21ed1a8e26da25c2495de3e32daf17ddcff9c4699ca8bfa6b5d01da2d9997a9f Nov 01 07:30:15.413677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.413646 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" event={"ID":"402eb62b-368e-447c-be22-a1e68493ad7c","Type":"ContainerStarted","Data":"21ed1a8e26da25c2495de3e32daf17ddcff9c4699ca8bfa6b5d01da2d9997a9f"} Nov 01 07:30:15.533127 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:15.533052 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z"] Nov 01 07:30:15.535227 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:30:15.535200 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5403c5_bc63_43e7_8702_eebf3ffd6cd9.slice/crio-58c6cc36812efe44ae0b768e53d322cee2a45a5b10db70be538f402873a4dca8 WatchSource:0}: Error finding container 58c6cc36812efe44ae0b768e53d322cee2a45a5b10db70be538f402873a4dca8: Status 404 returned error can't find the container with id 58c6cc36812efe44ae0b768e53d322cee2a45a5b10db70be538f402873a4dca8 Nov 01 07:30:16.419847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.419804 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" event={"ID":"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9","Type":"ContainerStarted","Data":"64782b40eaf7640c1358b2e0bb207954d7e6202343a9c94c2c879bba6370941e"} Nov 01 07:30:16.420269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.419894 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:16.420269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.419918 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" event={"ID":"ec5403c5-bc63-43e7-8702-eebf3ffd6cd9","Type":"ContainerStarted","Data":"58c6cc36812efe44ae0b768e53d322cee2a45a5b10db70be538f402873a4dca8"} Nov 01 07:30:16.421379 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.421348 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" event={"ID":"402eb62b-368e-447c-be22-a1e68493ad7c","Type":"ContainerStarted","Data":"c886ad93fcf195a06abe2ffdee07cb31ac574a7055a60885050301672b1beef1"} Nov 01 07:30:16.421479 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.421421 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:16.532783 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.532739 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" podStartSLOduration=2.532725927 podStartE2EDuration="2.532725927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:30:16.53171973 +0000 UTC m=+4832.084508924" watchObservedRunningTime="2026-11-01 07:30:16.532725927 +0000 UTC m=+4832.085515120" Nov 01 07:30:16.643711 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:16.643666 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" podStartSLOduration=2.643655657 podStartE2EDuration="2.643655657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:30:16.642744917 +0000 UTC m=+4832.195534109" watchObservedRunningTime="2026-11-01 07:30:16.643655657 +0000 UTC m=+4832.196444849" Nov 01 07:30:17.516613 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.516585 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" podUID="f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" containerName="payload-processing" containerID="cri-o://1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4" gracePeriod=25 Nov 01 07:30:17.757133 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.757111 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 07:30:17.839526 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.839468 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume\") pod \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " Nov 01 07:30:17.839526 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.839510 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98\" (UniqueName: \"kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98\") pod \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\" (UID: \"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124\") " Nov 01 07:30:17.839796 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.839775 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume" pod "f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" (UID: "f72e4bf9-85e3-4aff-b5ff-f98bcbe80124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:30:17.841721 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.841694 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98" pod "f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" (UID: "f72e4bf9-85e3-4aff-b5ff-f98bcbe80124"). InnerVolumeSpecName "kube-api-access-mmf98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:30:17.940819 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.940795 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:17.940819 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:17.940819 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmf98\" (UniqueName: \"kubernetes.io/projected/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124-kube-api-access-mmf98\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:18.433120 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.433087 4914 generic.go:356] "Generic (PLEG): container finished" podID="f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" containerID="1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4" exitCode=0 Nov 01 07:30:18.433269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.433149 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" Nov 01 07:30:18.433269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.433176 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" event={"ID":"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124","Type":"ContainerDied","Data":"1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4"} Nov 01 07:30:18.433269 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.433255 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-q2fb6" event={"ID":"f72e4bf9-85e3-4aff-b5ff-f98bcbe80124","Type":"ContainerDied","Data":"ff9ece1f5ad6dc478e379a6c2052e7694db9963a668962c922a1c39ac2b647ba"} Nov 01 07:30:18.433436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.433301 4914 scope.go:122] "RemoveContainer" containerID="1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4" Nov 01 07:30:18.442148 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.442130 4914 scope.go:122] "RemoveContainer" containerID="1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4" Nov 01 07:30:18.442467 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:30:18.442448 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4\": container with ID starting with 1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4 not found: ID does not exist" containerID="1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4" Nov 01 07:30:18.442545 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.442477 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4"} err="failed to get container status \"1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4\": rpc error: code = NotFound desc = could not find container \"1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4\": container with ID starting with 1108d7fbd4a53cceb55fccdbabcd97011ec35851f61ec6b98883412ee84fb1f4 not found: ID does not exist" Nov 01 07:30:18.593847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.593824 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-q2fb6"] Nov 01 07:30:18.643319 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:18.643301 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-q2fb6"] Nov 01 07:30:20.615411 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:20.615382 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" path="/var/lib/kubelet/pods/f72e4bf9-85e3-4aff-b5ff-f98bcbe80124/volumes" Nov 01 07:30:27.429195 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:27.429167 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5b4cf47bfb-xjj8z" Nov 01 07:30:27.429574 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:27.429232 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6759564b6c-wpg92" Nov 01 07:30:27.567120 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:27.567088 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-54ddd7df84-r5p2d"] Nov 01 07:30:27.679789 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:27.679713 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-56f8865864-tcnsn"] Nov 01 07:30:32.568070 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.568031 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" podUID="b949b69f-92fc-4690-9f87-e3b54f6e515a" containerName="payload-processing" containerID="cri-o://b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180" gracePeriod=25 Nov 01 07:30:32.680966 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.680932 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" podUID="d946dd9f-2e33-4fea-9607-1d7be53372aa" containerName="payload-pre-processing" containerID="cri-o://8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a" gracePeriod=25 Nov 01 07:30:32.827969 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.827914 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:32.964924 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.964869 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:30:32.968881 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.968840 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj\" (UniqueName: \"kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj\") pod \"b949b69f-92fc-4690-9f87-e3b54f6e515a\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " Nov 01 07:30:32.969051 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.968928 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume\") pod \"b949b69f-92fc-4690-9f87-e3b54f6e515a\" (UID: \"b949b69f-92fc-4690-9f87-e3b54f6e515a\") " Nov 01 07:30:32.969294 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.969259 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume" pod "b949b69f-92fc-4690-9f87-e3b54f6e515a" (UID: "b949b69f-92fc-4690-9f87-e3b54f6e515a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:30:32.971114 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:32.971088 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj" pod "b949b69f-92fc-4690-9f87-e3b54f6e515a" (UID: "b949b69f-92fc-4690-9f87-e3b54f6e515a"). InnerVolumeSpecName "kube-api-access-5fgkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:30:33.069981 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.069953 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume\") pod \"d946dd9f-2e33-4fea-9607-1d7be53372aa\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " Nov 01 07:30:33.070160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.070013 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9\" (UniqueName: \"kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9\") pod \"d946dd9f-2e33-4fea-9607-1d7be53372aa\" (UID: \"d946dd9f-2e33-4fea-9607-1d7be53372aa\") " Nov 01 07:30:33.070254 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.070237 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b949b69f-92fc-4690-9f87-e3b54f6e515a-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:33.070325 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.070263 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgkj\" (UniqueName: \"kubernetes.io/projected/b949b69f-92fc-4690-9f87-e3b54f6e515a-kube-api-access-5fgkj\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:33.070431 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.070406 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume" pod "d946dd9f-2e33-4fea-9607-1d7be53372aa" (UID: "d946dd9f-2e33-4fea-9607-1d7be53372aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:30:33.072214 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.072191 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9" pod "d946dd9f-2e33-4fea-9607-1d7be53372aa" (UID: "d946dd9f-2e33-4fea-9607-1d7be53372aa"). InnerVolumeSpecName "kube-api-access-5wtw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:30:33.171539 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.171513 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d946dd9f-2e33-4fea-9607-1d7be53372aa-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:33.171539 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.171538 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wtw9\" (UniqueName: \"kubernetes.io/projected/d946dd9f-2e33-4fea-9607-1d7be53372aa-kube-api-access-5wtw9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:30:33.502541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.502451 4914 generic.go:356] "Generic (PLEG): container finished" podID="d946dd9f-2e33-4fea-9607-1d7be53372aa" containerID="8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a" exitCode=0 Nov 01 07:30:33.502541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.502503 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" Nov 01 07:30:33.502541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.502532 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" event={"ID":"d946dd9f-2e33-4fea-9607-1d7be53372aa","Type":"ContainerDied","Data":"8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a"} Nov 01 07:30:33.502833 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.502581 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-56f8865864-tcnsn" event={"ID":"d946dd9f-2e33-4fea-9607-1d7be53372aa","Type":"ContainerDied","Data":"2889392875ca71babf165939a4032b223ab445fd546eb5b3ddc362112c2102b2"} Nov 01 07:30:33.502833 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.502606 4914 scope.go:122] "RemoveContainer" containerID="8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a" Nov 01 07:30:33.504118 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.504093 4914 generic.go:356] "Generic (PLEG): container finished" podID="b949b69f-92fc-4690-9f87-e3b54f6e515a" containerID="b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180" exitCode=0 Nov 01 07:30:33.504245 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.504130 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" Nov 01 07:30:33.504245 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.504195 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" event={"ID":"b949b69f-92fc-4690-9f87-e3b54f6e515a","Type":"ContainerDied","Data":"b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180"} Nov 01 07:30:33.504245 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.504236 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-54ddd7df84-r5p2d" event={"ID":"b949b69f-92fc-4690-9f87-e3b54f6e515a","Type":"ContainerDied","Data":"18595163f1d9a393a50aadef985aa082fabcf3dcaf5d0e655e1f6faf020cd721"} Nov 01 07:30:33.512251 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.512189 4914 scope.go:122] "RemoveContainer" containerID="8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a" Nov 01 07:30:33.512549 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:30:33.512528 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a\": container with ID starting with 8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a not found: ID does not exist" containerID="8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a" Nov 01 07:30:33.512650 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.512554 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a"} err="failed to get container status \"8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a\": rpc error: code = NotFound desc = could not find container \"8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a\": container with ID starting with 8bbec7ccd2214635efde74deba308ed0b2f7109ac17baa02e822d670d88cf78a not found: ID does not exist" Nov 01 07:30:33.512650 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.512572 4914 scope.go:122] "RemoveContainer" containerID="b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180" Nov 01 07:30:33.520212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.520195 4914 scope.go:122] "RemoveContainer" containerID="b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180" Nov 01 07:30:33.520463 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:30:33.520444 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180\": container with ID starting with b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180 not found: ID does not exist" containerID="b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180" Nov 01 07:30:33.520519 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.520469 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180"} err="failed to get container status \"b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180\": rpc error: code = NotFound desc = could not find container \"b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180\": container with ID starting with b7465bae615b0f564f306c9030a61f2850849ca9aa8a36518cc751133b3dd180 not found: ID does not exist" Nov 01 07:30:33.665356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.665336 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-54ddd7df84-r5p2d"] Nov 01 07:30:33.714829 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.714809 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-54ddd7df84-r5p2d"] Nov 01 07:30:33.758894 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.758826 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s"] Nov 01 07:30:33.759336 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759318 4914 cpu_manager.go:414] "containerMap: removing container" podUID="b949b69f-92fc-4690-9f87-e3b54f6e515a" containerName="payload-processing" Nov 01 07:30:33.759417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759339 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b949b69f-92fc-4690-9f87-e3b54f6e515a" containerName="payload-processing" Nov 01 07:30:33.759417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759360 4914 cpu_manager.go:414] "containerMap: removing container" podUID="f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" containerName="payload-processing" Nov 01 07:30:33.759417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759366 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" containerName="payload-processing" Nov 01 07:30:33.759417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759408 4914 cpu_manager.go:414] "containerMap: removing container" podUID="d946dd9f-2e33-4fea-9607-1d7be53372aa" containerName="payload-pre-processing" Nov 01 07:30:33.759417 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759418 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d946dd9f-2e33-4fea-9607-1d7be53372aa" containerName="payload-pre-processing" Nov 01 07:30:33.759609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759501 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b949b69f-92fc-4690-9f87-e3b54f6e515a" containerName="payload-processing" Nov 01 07:30:33.759609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759512 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72e4bf9-85e3-4aff-b5ff-f98bcbe80124" containerName="payload-processing" Nov 01 07:30:33.759609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.759531 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d946dd9f-2e33-4fea-9607-1d7be53372aa" containerName="payload-pre-processing" Nov 01 07:30:33.763961 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.763946 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.778111 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.778093 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-56f8865864-tcnsn"] Nov 01 07:30:33.784221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.784198 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-56f8865864-tcnsn"] Nov 01 07:30:33.817056 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.817038 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 07:30:33.817164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.817047 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 07:30:33.817164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.817107 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 07:30:33.820096 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.820067 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s"] Nov 01 07:30:33.826066 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.826046 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 07:30:33.861414 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.861388 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 01 07:30:33.861517 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.861469 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6tbxj\"" Nov 01 07:30:33.878698 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.878674 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.878786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.878725 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.878786 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.878742 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5vt\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.878859 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.878829 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.979731 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.979709 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.979849 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.979794 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.979849 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.979823 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5vt\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.979963 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.979858 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.980567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.980543 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.980642 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.980605 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:33.982357 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:33.982326 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:34.037095 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:34.037049 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5vt\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt\") pod \"payload-processing-external-model-5bcd7997-lbz5s\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:34.076978 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:34.076960 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:34.474330 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:34.474306 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s"] Nov 01 07:30:34.476781 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:30:34.476754 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b62498_d74e_41d2_9036_49a752d869f0.slice/crio-18bc3bd9b9e7dd44ad8601a56e2297d9ca031eafc0eb37c874d712f9b5041251 WatchSource:0}: Error finding container 18bc3bd9b9e7dd44ad8601a56e2297d9ca031eafc0eb37c874d712f9b5041251: Status 404 returned error can't find the container with id 18bc3bd9b9e7dd44ad8601a56e2297d9ca031eafc0eb37c874d712f9b5041251 Nov 01 07:30:34.510362 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:34.510335 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" event={"ID":"c5b62498-d74e-41d2-9036-49a752d869f0","Type":"ContainerStarted","Data":"18bc3bd9b9e7dd44ad8601a56e2297d9ca031eafc0eb37c874d712f9b5041251"} Nov 01 07:30:34.615048 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:34.615026 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b949b69f-92fc-4690-9f87-e3b54f6e515a" path="/var/lib/kubelet/pods/b949b69f-92fc-4690-9f87-e3b54f6e515a/volumes" Nov 01 07:30:34.615389 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:34.615376 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d946dd9f-2e33-4fea-9607-1d7be53372aa" path="/var/lib/kubelet/pods/d946dd9f-2e33-4fea-9607-1d7be53372aa/volumes" Nov 01 07:30:35.520031 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:35.519985 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" event={"ID":"c5b62498-d74e-41d2-9036-49a752d869f0","Type":"ContainerStarted","Data":"60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf"} Nov 01 07:30:35.520596 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:35.520143 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:30:35.634989 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:35.634927 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" podStartSLOduration=2.634909262 podStartE2EDuration="2.634909262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:30:35.631407372 +0000 UTC m=+4851.184196565" watchObservedRunningTime="2026-11-01 07:30:35.634909262 +0000 UTC m=+4851.187698454" Nov 01 07:30:37.733735 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:30:37.733683 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:30:46.537070 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:30:46.537043 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:31:07.592354 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:31:07.592318 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:31:37.592359 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:31:37.592267 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:32:07.592447 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:32:07.592408 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:32:27.020760 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:27.020725 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s"] Nov 01 07:32:31.799300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:31.798394 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:32:31.799300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:31.798493 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:32:32.021393 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.021334 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" podUID="c5b62498-d74e-41d2-9036-49a752d869f0" containerName="payload-processing" containerID="cri-o://60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf" gracePeriod=25 Nov 01 07:32:32.469979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.469949 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:32:32.540557 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.540533 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume\") pod \"c5b62498-d74e-41d2-9036-49a752d869f0\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " Nov 01 07:32:32.540663 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.540566 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume\") pod \"c5b62498-d74e-41d2-9036-49a752d869f0\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " Nov 01 07:32:32.540663 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.540660 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials\") pod \"c5b62498-d74e-41d2-9036-49a752d869f0\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " Nov 01 07:32:32.540773 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.540695 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt\") pod \"c5b62498-d74e-41d2-9036-49a752d869f0\" (UID: \"c5b62498-d74e-41d2-9036-49a752d869f0\") " Nov 01 07:32:32.540978 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.540952 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume" pod "c5b62498-d74e-41d2-9036-49a752d869f0" (UID: "c5b62498-d74e-41d2-9036-49a752d869f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:32:32.541060 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.540983 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume" pod "c5b62498-d74e-41d2-9036-49a752d869f0" (UID: "c5b62498-d74e-41d2-9036-49a752d869f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:32:32.542932 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.542912 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt" pod "c5b62498-d74e-41d2-9036-49a752d869f0" (UID: "c5b62498-d74e-41d2-9036-49a752d869f0"). InnerVolumeSpecName "kube-api-access-lp5vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:32:32.542983 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.542960 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials" pod "c5b62498-d74e-41d2-9036-49a752d869f0" (UID: "c5b62498-d74e-41d2-9036-49a752d869f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:32:32.641644 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.641588 4914 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-provider-credentials\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:32:32.641644 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.641611 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp5vt\" (UniqueName: \"kubernetes.io/projected/c5b62498-d74e-41d2-9036-49a752d869f0-kube-api-access-lp5vt\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:32:32.641644 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.641627 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:32:32.641644 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:32.641638 4914 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5b62498-d74e-41d2-9036-49a752d869f0-routing-overlay-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:32:33.105623 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.105596 4914 generic.go:356] "Generic (PLEG): container finished" podID="c5b62498-d74e-41d2-9036-49a752d869f0" containerID="60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf" exitCode=0 Nov 01 07:32:33.106037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.105643 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" Nov 01 07:32:33.106037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.105677 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" event={"ID":"c5b62498-d74e-41d2-9036-49a752d869f0","Type":"ContainerDied","Data":"60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf"} Nov 01 07:32:33.106037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.105730 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s" event={"ID":"c5b62498-d74e-41d2-9036-49a752d869f0","Type":"ContainerDied","Data":"18bc3bd9b9e7dd44ad8601a56e2297d9ca031eafc0eb37c874d712f9b5041251"} Nov 01 07:32:33.106037 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.105757 4914 scope.go:122] "RemoveContainer" containerID="60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf" Nov 01 07:32:33.114423 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.114393 4914 scope.go:122] "RemoveContainer" containerID="60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf" Nov 01 07:32:33.114696 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:32:33.114674 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf\": container with ID starting with 60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf not found: ID does not exist" containerID="60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf" Nov 01 07:32:33.114737 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.114710 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf"} err="failed to get container status \"60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf\": rpc error: code = NotFound desc = could not find container \"60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf\": container with ID starting with 60bdde9774ccba97e15c11b7ccb4daf6bc7f43021b246145f4ec2c36ff5bcadf not found: ID does not exist" Nov 01 07:32:33.268182 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.268158 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s"] Nov 01 07:32:33.269745 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:33.269725 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lbz5s"] Nov 01 07:32:34.616866 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:32:34.616830 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b62498-d74e-41d2-9036-49a752d869f0" path="/var/lib/kubelet/pods/c5b62498-d74e-41d2-9036-49a752d869f0/volumes" Nov 01 07:32:37.595797 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:32:37.595758 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:33:07.595660 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:33:07.595573 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:33:18.062423 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:33:18.062362 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-krv87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:33:18.063171 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:33:18.063117 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-krv87" podUID="430c5975-a53c-4dde-814a-6661ad4e71b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:33:37.594142 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:33:37.594103 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:34:07.595712 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:34:07.595670 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:34:37.606001 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:34:37.605895 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:34:45.230907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:34:45.230877 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:34:45.254165 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:34:45.254141 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:35:00.725414 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:35:00.725373 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:35:00.726014 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:35:00.725980 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:35:07.603934 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:35:07.603890 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:35:37.591265 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:35:37.591231 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:36:07.590849 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:36:07.590765 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:36:37.591344 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:36:37.591308 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:37:04.231061 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.231027 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj"] Nov 01 07:37:04.231672 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.231615 4914 cpu_manager.go:414] "containerMap: removing container" podUID="c5b62498-d74e-41d2-9036-49a752d869f0" containerName="payload-processing" Nov 01 07:37:04.231672 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.231636 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b62498-d74e-41d2-9036-49a752d869f0" containerName="payload-processing" Nov 01 07:37:04.231804 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.231784 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b62498-d74e-41d2-9036-49a752d869f0" containerName="payload-processing" Nov 01 07:37:04.234796 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.234773 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.316580 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.316554 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2624b0-openshift-default-dockercfg-bvwxf\"" Nov 01 07:37:04.338012 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.337988 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj"] Nov 01 07:37:04.366137 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366110 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366165 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366302 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366265 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366421 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366375 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwt9\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366473 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366451 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366524 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366504 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366572 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366559 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366612 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366583 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.366612 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.366600 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467466 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467427 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467466 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467471 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467490 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467511 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467533 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467603 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467715 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467647 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467974 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467762 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467974 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467829 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwt9\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467974 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467900 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.467974 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.467962 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.468181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.468087 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.468181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.468141 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.468460 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.468442 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.470072 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.470054 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.470153 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.470133 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.528314 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.528221 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.528507 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.528478 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwt9\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9\") pod \"e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.547471 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.547445 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:04.964750 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.964577 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj"] Nov 01 07:37:04.967480 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:37:04.967452 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda07c0c_7263_4923_bfed_a75ed51b9349.slice/crio-3af206e17f930d8e06414988b14f21092740f4be3eff82044f2dd9dc4ec4d2ae WatchSource:0}: Error finding container 3af206e17f930d8e06414988b14f21092740f4be3eff82044f2dd9dc4ec4d2ae: Status 404 returned error can't find the container with id 3af206e17f930d8e06414988b14f21092740f4be3eff82044f2dd9dc4ec4d2ae Nov 01 07:37:04.969392 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.969359 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:37:04.969465 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.969422 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:37:04.969465 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:04.969450 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:37:05.853298 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:05.853243 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" event={"ID":"dda07c0c-7263-4923-bfed-a75ed51b9349","Type":"ContainerStarted","Data":"d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862"} Nov 01 07:37:05.853782 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:05.853326 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" event={"ID":"dda07c0c-7263-4923-bfed-a75ed51b9349","Type":"ContainerStarted","Data":"3af206e17f930d8e06414988b14f21092740f4be3eff82044f2dd9dc4ec4d2ae"} Nov 01 07:37:06.019160 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:06.019114 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podStartSLOduration=2.019100572 podStartE2EDuration="2.019100572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:37:06.016801828 +0000 UTC m=+5241.569591039" watchObservedRunningTime="2026-11-01 07:37:06.019100572 +0000 UTC m=+5241.571889764" Nov 01 07:37:06.547688 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:06.547660 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:06.552723 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:06.552694 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:06.858712 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:06.858685 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:06.859666 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:06.859648 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:07.594467 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:37:07.594433 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:37:12.525593 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.525550 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx"] Nov 01 07:37:12.530406 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.530388 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.585579 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.585557 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx"] Nov 01 07:37:12.633514 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.633465 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2624b0\"" Nov 01 07:37:12.633840 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.633819 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fda44255-dockercfg-dgjvt\"" Nov 01 07:37:12.668973 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.668929 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7rk\" (UniqueName: \"kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk\") pod \"payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.669175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.669015 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.685079 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.685048 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl"] Nov 01 07:37:12.688713 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.688692 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:12.741527 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.741493 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl"] Nov 01 07:37:12.782868 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.781922 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.782868 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.782199 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7rk\" (UniqueName: \"kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk\") pod \"payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.783136 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.783110 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.838557 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.838531 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7rk\" (UniqueName: \"kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk\") pod \"payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.843423 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.843396 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:12.883452 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.883344 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nhg\" (UniqueName: \"kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg\") pod \"payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:12.883591 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.883524 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:12.984435 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.984407 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:12.984584 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.984484 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nhg\" (UniqueName: \"kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg\") pod \"payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:12.985111 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:12.985091 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:13.040794 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.040730 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nhg\" (UniqueName: \"kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg\") pod \"payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:13.210159 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.210135 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx"] Nov 01 07:37:13.211856 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:37:13.211829 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62e0826_8903_4e30_b846_a9d4b637d1d9.slice/crio-3f91d43bd5303e4d284899253074f4d52738eec645c593fdc3c31978f392944e WatchSource:0}: Error finding container 3f91d43bd5303e4d284899253074f4d52738eec645c593fdc3c31978f392944e: Status 404 returned error can't find the container with id 3f91d43bd5303e4d284899253074f4d52738eec645c593fdc3c31978f392944e Nov 01 07:37:13.301434 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.301389 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:13.669914 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.669890 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl"] Nov 01 07:37:13.671878 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:37:13.671849 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413929ac_9911_4142_b96d_be2c2d293a52.slice/crio-4b51655d9893236a0780e515073ad88dd81e52e045cde8adf2aff31a25dd89ba WatchSource:0}: Error finding container 4b51655d9893236a0780e515073ad88dd81e52e045cde8adf2aff31a25dd89ba: Status 404 returned error can't find the container with id 4b51655d9893236a0780e515073ad88dd81e52e045cde8adf2aff31a25dd89ba Nov 01 07:37:13.890426 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.890397 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" event={"ID":"413929ac-9911-4142-b96d-be2c2d293a52","Type":"ContainerStarted","Data":"51f4369015c36902bac6e2b082e68cfa37437a8e179d77846d15d976ae56fcc1"} Nov 01 07:37:13.890577 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.890458 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" event={"ID":"413929ac-9911-4142-b96d-be2c2d293a52","Type":"ContainerStarted","Data":"4b51655d9893236a0780e515073ad88dd81e52e045cde8adf2aff31a25dd89ba"} Nov 01 07:37:13.890654 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.890625 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:13.892018 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.891982 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" event={"ID":"d62e0826-8903-4e30-b846-a9d4b637d1d9","Type":"ContainerStarted","Data":"6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525"} Nov 01 07:37:13.892134 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.892030 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" event={"ID":"d62e0826-8903-4e30-b846-a9d4b637d1d9","Type":"ContainerStarted","Data":"3f91d43bd5303e4d284899253074f4d52738eec645c593fdc3c31978f392944e"} Nov 01 07:37:13.892134 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:13.892097 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:14.006842 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.006768 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" podStartSLOduration=2.006755817 podStartE2EDuration="2.006755817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:37:14.004478577 +0000 UTC m=+5249.557267795" watchObservedRunningTime="2026-11-01 07:37:14.006755817 +0000 UTC m=+5249.559545146" Nov 01 07:37:14.129098 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.129031 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" podStartSLOduration=2.129016424 podStartE2EDuration="2.129016424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:37:14.126211166 +0000 UTC m=+5249.679000371" watchObservedRunningTime="2026-11-01 07:37:14.129016424 +0000 UTC m=+5249.681805620" Nov 01 07:37:14.869677 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.869642 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc"] Nov 01 07:37:14.873423 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.873398 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:14.901951 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.901925 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:14.902107 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.901991 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68lg\" (UniqueName: \"kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:14.902329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.902305 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:14.902478 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.902459 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:14.927229 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.927208 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2624b0-serving-cert\"" Nov 01 07:37:14.979912 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:14.979881 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc"] Nov 01 07:37:15.003565 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.003539 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.003676 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.003583 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.003676 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.003650 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68lg\" (UniqueName: \"kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.003797 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.003696 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.003797 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:37:15.003719 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-serving-cert: secret "maas-api-e2e-worker-single-main-2624b0-serving-cert" not found Nov 01 07:37:15.003905 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:37:15.003801 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls podName:4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4 nodeName:}" failed. No retries permitted until 2026-11-01 07:37:15.503779132 +0000 UTC m=+5251.056568323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls") pod "maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" (UID: "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4") : secret "maas-api-e2e-worker-single-main-2624b0-serving-cert" not found Nov 01 07:37:15.007321 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.004366 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.007321 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.006967 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.062083 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.062062 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68lg\" (UniqueName: \"kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.508686 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.508654 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.511286 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.511253 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:15.790671 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:15.790593 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:16.168683 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:16.168657 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc"] Nov 01 07:37:16.170429 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:37:16.170404 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5fc7cd_5ef5_49ce_ab7f_ac613552c1f4.slice/crio-08dcff7ebfb79a8a7b45949253e2d97cd70c253d2f755d3a289ba1c794c3c4ce WatchSource:0}: Error finding container 08dcff7ebfb79a8a7b45949253e2d97cd70c253d2f755d3a289ba1c794c3c4ce: Status 404 returned error can't find the container with id 08dcff7ebfb79a8a7b45949253e2d97cd70c253d2f755d3a289ba1c794c3c4ce Nov 01 07:37:16.172056 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:16.172035 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:37:16.925076 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:16.925010 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" event={"ID":"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4","Type":"ContainerStarted","Data":"08dcff7ebfb79a8a7b45949253e2d97cd70c253d2f755d3a289ba1c794c3c4ce"} Nov 01 07:37:17.931636 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:17.931599 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" event={"ID":"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4","Type":"ContainerStarted","Data":"91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa"} Nov 01 07:37:17.932025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:17.931704 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:18.044344 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:18.044291 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" podStartSLOduration=3.177367188 podStartE2EDuration="4.044259414s" totalImagesPullingTime="866.892226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:37:18.043440252 +0000 UTC m=+5253.596229445" watchObservedRunningTime="2026-11-01 07:37:18.044259414 +0000 UTC m=+5253.597048606" Nov 01 07:37:23.942832 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:23.942799 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:37:24.899907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:24.899874 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:37:24.900102 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:24.900067 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:37:32.013350 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:32.013262 4914 generic.go:356] "Generic (PLEG): container finished" podID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerID="d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862" exitCode=0 Nov 01 07:37:32.013691 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:32.013338 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" event={"ID":"dda07c0c-7263-4923-bfed-a75ed51b9349","Type":"ContainerDied","Data":"d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862"} Nov 01 07:37:32.013691 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:32.013660 4914 scope.go:122] "RemoveContainer" containerID="d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862" Nov 01 07:37:32.014852 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:32.014813 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:37:32.014950 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:32.014894 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:37:32.014950 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:32.014924 4914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 07:37:33.020288 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:33.020242 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" event={"ID":"dda07c0c-7263-4923-bfed-a75ed51b9349","Type":"ContainerStarted","Data":"b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd"} Nov 01 07:37:33.563300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:33.559375 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:33.563300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:33.559491 4914 patch_prober.go:29] interesting pod/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 07:37:33.563300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:33.559531 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 07:37:34.565301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:34.550672 4914 patch_prober.go:29] interesting pod/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 07:37:34.565301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:34.550744 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 07:37:35.568299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:35.566364 4914 patch_prober.go:29] interesting pod/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 07:37:35.568299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:35.566434 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 07:37:36.571300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:36.569258 4914 patch_prober.go:29] interesting pod/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 07:37:36.571300 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:36.569336 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 07:37:36.871346 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:36.866359 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:37.549727 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:37.549691 4914 patch_prober.go:29] interesting pod/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 01 07:37:37.551646 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:37.551613 4914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 01 07:37:37.599077 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:37:37.599046 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:37:38.552032 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:38.552001 4914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:37:38.553088 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:37:38.553067 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:38:07.595821 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:38:07.595781 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:38:37.591035 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:38:37.591001 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:38:39.275104 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.275068 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl"] Nov 01 07:38:39.380008 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.379973 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc"] Nov 01 07:38:39.380222 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.380177 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" podUID="4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" containerName="maas-api" containerID="cri-o://91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa" gracePeriod=30 Nov 01 07:38:39.382810 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.382781 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx"] Nov 01 07:38:39.823688 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.823664 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:38:39.965421 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.965350 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\") pod \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " Nov 01 07:38:39.965421 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.965398 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs\") pod \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " Nov 01 07:38:39.965664 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.965477 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca\") pod \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " Nov 01 07:38:39.965664 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.965543 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg\" (UniqueName: \"kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg\") pod \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\" (UID: \"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4\") " Nov 01 07:38:39.965906 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.965877 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca" pod "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" (UID: "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:38:39.967812 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.967775 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs" pod "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" (UID: "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:38:39.967932 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.967831 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls" pod "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" (UID: "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:38:39.968244 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:39.968220 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg" pod "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" (UID: "4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4"). InnerVolumeSpecName "kube-api-access-x68lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:38:40.066655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.066633 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:40.066759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.066660 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68lg\" (UniqueName: \"kubernetes.io/projected/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-kube-api-access-x68lg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:40.066759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.066677 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:40.066759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.066689 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:38:40.359678 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.359632 4914 generic.go:356] "Generic (PLEG): container finished" podID="4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" containerID="91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa" exitCode=0 Nov 01 07:38:40.360263 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.359795 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" event={"ID":"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4","Type":"ContainerDied","Data":"91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa"} Nov 01 07:38:40.360263 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.359878 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" event={"ID":"4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4","Type":"ContainerDied","Data":"08dcff7ebfb79a8a7b45949253e2d97cd70c253d2f755d3a289ba1c794c3c4ce"} Nov 01 07:38:40.360263 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.359907 4914 scope.go:122] "RemoveContainer" containerID="91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa" Nov 01 07:38:40.360263 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.360212 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc" Nov 01 07:38:40.387447 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.387312 4914 scope.go:122] "RemoveContainer" containerID="91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa" Nov 01 07:38:40.388767 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:38:40.388234 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa\": container with ID starting with 91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa not found: ID does not exist" containerID="91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa" Nov 01 07:38:40.388997 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.388956 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa"} err="failed to get container status \"91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa\": rpc error: code = NotFound desc = could not find container \"91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa\": container with ID starting with 91988cc7aa34e56450697322637b23d7ba40b0db4d57f0efc538bc51e0489aaa not found: ID does not exist" Nov 01 07:38:40.527911 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.527844 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc"] Nov 01 07:38:40.530799 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.530755 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-2lnvc"] Nov 01 07:38:40.615971 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:40.615901 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" path="/var/lib/kubelet/pods/4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4/volumes" Nov 01 07:38:44.275962 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:44.275928 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" podUID="413929ac-9911-4142-b96d-be2c2d293a52" containerName="payload-processing" containerID="cri-o://51f4369015c36902bac6e2b082e68cfa37437a8e179d77846d15d976ae56fcc1" gracePeriod=25 Nov 01 07:38:44.383493 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:44.383461 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerName="payload-pre-processing" containerID="cri-o://6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525" gracePeriod=25 Nov 01 07:38:44.899013 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:44.898973 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:38:44.899196 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:44.898987 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" podUID="413929ac-9911-4142-b96d-be2c2d293a52" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:38:54.898665 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:54.898628 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:38:54.898665 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:38:54.898659 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" podUID="413929ac-9911-4142-b96d-be2c2d293a52" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 07:39:04.470745 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.470711 4914 generic.go:356] "Generic (PLEG): container finished" podID="413929ac-9911-4142-b96d-be2c2d293a52" containerID="51f4369015c36902bac6e2b082e68cfa37437a8e179d77846d15d976ae56fcc1" exitCode=0 Nov 01 07:39:04.471161 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.470790 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" event={"ID":"413929ac-9911-4142-b96d-be2c2d293a52","Type":"ContainerDied","Data":"51f4369015c36902bac6e2b082e68cfa37437a8e179d77846d15d976ae56fcc1"} Nov 01 07:39:04.532758 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.532685 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:39:04.577198 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.577166 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg\" (UniqueName: \"kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg\") pod \"413929ac-9911-4142-b96d-be2c2d293a52\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " Nov 01 07:39:04.577323 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.577262 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume\") pod \"413929ac-9911-4142-b96d-be2c2d293a52\" (UID: \"413929ac-9911-4142-b96d-be2c2d293a52\") " Nov 01 07:39:04.577609 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.577571 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume" pod "413929ac-9911-4142-b96d-be2c2d293a52" (UID: "413929ac-9911-4142-b96d-be2c2d293a52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:39:04.577726 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.577704 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/413929ac-9911-4142-b96d-be2c2d293a52-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:04.579361 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.579338 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg" pod "413929ac-9911-4142-b96d-be2c2d293a52" (UID: "413929ac-9911-4142-b96d-be2c2d293a52"). InnerVolumeSpecName "kube-api-access-n9nhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:39:04.630515 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.630497 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:39:04.678920 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.678894 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk\" (UniqueName: \"kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk\") pod \"d62e0826-8903-4e30-b846-a9d4b637d1d9\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " Nov 01 07:39:04.679067 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.679050 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume\") pod \"d62e0826-8903-4e30-b846-a9d4b637d1d9\" (UID: \"d62e0826-8903-4e30-b846-a9d4b637d1d9\") " Nov 01 07:39:04.679405 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.679383 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9nhg\" (UniqueName: \"kubernetes.io/projected/413929ac-9911-4142-b96d-be2c2d293a52-kube-api-access-n9nhg\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:04.679522 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.679451 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume" pod "d62e0826-8903-4e30-b846-a9d4b637d1d9" (UID: "d62e0826-8903-4e30-b846-a9d4b637d1d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:39:04.680979 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.680961 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk" pod "d62e0826-8903-4e30-b846-a9d4b637d1d9" (UID: "d62e0826-8903-4e30-b846-a9d4b637d1d9"). InnerVolumeSpecName "kube-api-access-wb7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:39:04.780685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.780663 4914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d62e0826-8903-4e30-b846-a9d4b637d1d9-plugins-config-volume\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:04.780685 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:04.780687 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7rk\" (UniqueName: \"kubernetes.io/projected/d62e0826-8903-4e30-b846-a9d4b637d1d9-kube-api-access-wb7rk\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:05.476860 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.476822 4914 generic.go:356] "Generic (PLEG): container finished" podID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerID="6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525" exitCode=0 Nov 01 07:39:05.477308 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.476910 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" Nov 01 07:39:05.477308 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.476924 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" event={"ID":"d62e0826-8903-4e30-b846-a9d4b637d1d9","Type":"ContainerDied","Data":"6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525"} Nov 01 07:39:05.477308 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.477014 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx" event={"ID":"d62e0826-8903-4e30-b846-a9d4b637d1d9","Type":"ContainerDied","Data":"3f91d43bd5303e4d284899253074f4d52738eec645c593fdc3c31978f392944e"} Nov 01 07:39:05.477308 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.477045 4914 scope.go:122] "RemoveContainer" containerID="6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525" Nov 01 07:39:05.478426 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.478411 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" Nov 01 07:39:05.478488 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.478422 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl" event={"ID":"413929ac-9911-4142-b96d-be2c2d293a52","Type":"ContainerDied","Data":"4b51655d9893236a0780e515073ad88dd81e52e045cde8adf2aff31a25dd89ba"} Nov 01 07:39:05.486728 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.486711 4914 scope.go:122] "RemoveContainer" containerID="6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525" Nov 01 07:39:05.486991 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:05.486974 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525\": container with ID starting with 6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525 not found: ID does not exist" containerID="6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525" Nov 01 07:39:05.487053 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.486999 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525"} err="failed to get container status \"6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525\": rpc error: code = NotFound desc = could not find container \"6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525\": container with ID starting with 6965594760f719ea2a50b8ea2e66d70805347ff65cc203ba0c2026a743ee8525 not found: ID does not exist" Nov 01 07:39:05.487053 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.487017 4914 scope.go:122] "RemoveContainer" containerID="51f4369015c36902bac6e2b082e68cfa37437a8e179d77846d15d976ae56fcc1" Nov 01 07:39:05.638513 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.638489 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl"] Nov 01 07:39:05.687612 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.687591 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2624b0-fc9d6d767lg2hl"] Nov 01 07:39:05.794976 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.794952 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx"] Nov 01 07:39:05.798403 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:05.798382 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2624b0-57979wndqx"] Nov 01 07:39:06.614796 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:06.614764 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413929ac-9911-4142-b96d-be2c2d293a52" path="/var/lib/kubelet/pods/413929ac-9911-4142-b96d-be2c2d293a52/volumes" Nov 01 07:39:06.615177 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:06.615104 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" path="/var/lib/kubelet/pods/d62e0826-8903-4e30-b846-a9d4b637d1d9/volumes" Nov 01 07:39:07.590516 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:07.590484 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:39:15.906882 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.906846 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8"] Nov 01 07:39:15.907332 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907267 4914 cpu_manager.go:414] "containerMap: removing container" podUID="413929ac-9911-4142-b96d-be2c2d293a52" containerName="payload-processing" Nov 01 07:39:15.907332 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907303 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413929ac-9911-4142-b96d-be2c2d293a52" containerName="payload-processing" Nov 01 07:39:15.907332 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907314 4914 cpu_manager.go:414] "containerMap: removing container" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerName="payload-pre-processing" Nov 01 07:39:15.907332 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907319 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerName="payload-pre-processing" Nov 01 07:39:15.907475 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907335 4914 cpu_manager.go:414] "containerMap: removing container" podUID="4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" containerName="maas-api" Nov 01 07:39:15.907475 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907340 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" containerName="maas-api" Nov 01 07:39:15.907475 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907405 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="413929ac-9911-4142-b96d-be2c2d293a52" containerName="payload-processing" Nov 01 07:39:15.907475 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907412 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62e0826-8903-4e30-b846-a9d4b637d1d9" containerName="payload-pre-processing" Nov 01 07:39:15.907475 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.907420 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5fc7cd-5ef5-49ce-ab7f-ac613552c1f4" containerName="maas-api" Nov 01 07:39:15.912089 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.912063 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:15.968164 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.968132 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2m9\" (UniqueName: \"kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:15.968329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.968193 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:15.968329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.968264 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:15.968329 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:15.968313 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.015267 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.015239 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2624b0-serving-cert\"" Nov 01 07:39:16.017539 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.017518 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8"] Nov 01 07:39:16.069347 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.069315 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.069470 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.069352 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.069470 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.069405 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2m9\" (UniqueName: \"kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.069470 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.069434 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.069657 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:16.069531 4914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-serving-cert: secret "maas-api-e2e-worker-single-main-2624b0-serving-cert" not found Nov 01 07:39:16.069657 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:16.069588 4914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls podName:1f439840-c085-4206-abfb-5b122ce41314 nodeName:}" failed. No retries permitted until 2026-11-01 07:39:16.569568208 +0000 UTC m=+5372.122357380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls") pod "maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" (UID: "1f439840-c085-4206-abfb-5b122ce41314") : secret "maas-api-e2e-worker-single-main-2624b0-serving-cert" not found Nov 01 07:39:16.070170 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.070142 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.071939 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.071916 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.128849 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.128814 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2m9\" (UniqueName: \"kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.573915 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.573882 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.576435 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.576406 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:16.825374 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:16.825301 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:17.196237 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:17.196211 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8"] Nov 01 07:39:17.197944 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:39:17.197906 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f439840_c085_4206_abfb_5b122ce41314.slice/crio-d6e3f924f10e0cfe122548ce53a70eb9ffaa5d5859b9c6597671622fbc4c14ef WatchSource:0}: Error finding container d6e3f924f10e0cfe122548ce53a70eb9ffaa5d5859b9c6597671622fbc4c14ef: Status 404 returned error can't find the container with id d6e3f924f10e0cfe122548ce53a70eb9ffaa5d5859b9c6597671622fbc4c14ef Nov 01 07:39:17.535960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:17.535901 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" event={"ID":"1f439840-c085-4206-abfb-5b122ce41314","Type":"ContainerStarted","Data":"d6e3f924f10e0cfe122548ce53a70eb9ffaa5d5859b9c6597671622fbc4c14ef"} Nov 01 07:39:18.543002 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:18.542966 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" event={"ID":"1f439840-c085-4206-abfb-5b122ce41314","Type":"ContainerStarted","Data":"3db2d414a3658da7d70f572c8672bede17e38ea3fead388e3d4d2fded25c7895"} Nov 01 07:39:18.543399 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:18.543034 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:18.656489 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:18.656436 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" podStartSLOduration=2.768868395 podStartE2EDuration="3.656419687s" totalImagesPullingTime="887.551292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:39:18.655538603 +0000 UTC m=+5374.208327796" watchObservedRunningTime="2026-11-01 07:39:18.656419687 +0000 UTC m=+5374.209208881" Nov 01 07:39:24.553800 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:24.553769 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:37.592141 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:37.592107 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:39:43.497618 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.497577 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8"] Nov 01 07:39:43.498099 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.497830 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" podUID="1f439840-c085-4206-abfb-5b122ce41314" containerName="maas-api" containerID="cri-o://3db2d414a3658da7d70f572c8672bede17e38ea3fead388e3d4d2fded25c7895" gracePeriod=30 Nov 01 07:39:43.663671 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.663631 4914 generic.go:356] "Generic (PLEG): container finished" podID="1f439840-c085-4206-abfb-5b122ce41314" containerID="3db2d414a3658da7d70f572c8672bede17e38ea3fead388e3d4d2fded25c7895" exitCode=0 Nov 01 07:39:43.663866 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.663712 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" event={"ID":"1f439840-c085-4206-abfb-5b122ce41314","Type":"ContainerDied","Data":"3db2d414a3658da7d70f572c8672bede17e38ea3fead388e3d4d2fded25c7895"} Nov 01 07:39:43.769758 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.769735 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:43.921212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.921178 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\") pod \"1f439840-c085-4206-abfb-5b122ce41314\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " Nov 01 07:39:43.921212 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.921214 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs\") pod \"1f439840-c085-4206-abfb-5b122ce41314\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " Nov 01 07:39:43.921492 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.921263 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca\") pod \"1f439840-c085-4206-abfb-5b122ce41314\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " Nov 01 07:39:43.921492 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.921324 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9\" (UniqueName: \"kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9\") pod \"1f439840-c085-4206-abfb-5b122ce41314\" (UID: \"1f439840-c085-4206-abfb-5b122ce41314\") " Nov 01 07:39:43.921753 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.921721 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca" pod "1f439840-c085-4206-abfb-5b122ce41314" (UID: "1f439840-c085-4206-abfb-5b122ce41314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:39:43.923738 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.923718 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs" pod "1f439840-c085-4206-abfb-5b122ce41314" (UID: "1f439840-c085-4206-abfb-5b122ce41314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:39:43.923847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.923792 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9" pod "1f439840-c085-4206-abfb-5b122ce41314" (UID: "1f439840-c085-4206-abfb-5b122ce41314"). InnerVolumeSpecName "kube-api-access-fg2m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:39:43.923847 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:43.923671 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls" pod "1f439840-c085-4206-abfb-5b122ce41314" (UID: "1f439840-c085-4206-abfb-5b122ce41314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:39:44.023025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.022947 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:44.023025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.022983 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f439840-c085-4206-abfb-5b122ce41314-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:44.023025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.022999 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f439840-c085-4206-abfb-5b122ce41314-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:44.023025 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.023014 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2m9\" (UniqueName: \"kubernetes.io/projected/1f439840-c085-4206-abfb-5b122ce41314-kube-api-access-fg2m9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:44.670551 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.670529 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" Nov 01 07:39:44.670907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.670558 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8" event={"ID":"1f439840-c085-4206-abfb-5b122ce41314","Type":"ContainerDied","Data":"d6e3f924f10e0cfe122548ce53a70eb9ffaa5d5859b9c6597671622fbc4c14ef"} Nov 01 07:39:44.670907 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.670614 4914 scope.go:122] "RemoveContainer" containerID="3db2d414a3658da7d70f572c8672bede17e38ea3fead388e3d4d2fded25c7895" Nov 01 07:39:44.831948 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.831923 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8"] Nov 01 07:39:44.881181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:44.881126 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2624b0-f487665f7-xcjj8"] Nov 01 07:39:45.261387 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:45.261328 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:39:45.284967 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:45.284944 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:39:45.933197 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:45.933167 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj"] Nov 01 07:39:45.933653 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:45.933394 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" containerID="cri-o://b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd" gracePeriod=30 Nov 01 07:39:46.615264 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:46.615231 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f439840-c085-4206-abfb-5b122ce41314" path="/var/lib/kubelet/pods/1f439840-c085-4206-abfb-5b122ce41314/volumes" Nov 01 07:39:51.181670 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.181646 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:39:51.304325 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304260 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304325 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304308 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304346 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304369 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304410 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304435 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304482 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304515 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:39:51.304751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304523 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304613 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:39:51.304751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304621 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs\") pod \"dda07c0c-7263-4923-bfed-a75ed51b9349\" (UID: \"dda07c0c-7263-4923-bfed-a75ed51b9349\") " Nov 01 07:39:51.304751 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304738 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:39:51.305056 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304781 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:39:51.305056 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.304905 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:39:51.305168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.305086 4914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.305168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.305107 4914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-credential-socket\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.305168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.305121 4914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-workload-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.305168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.305134 4914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-data\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.305168 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.305158 4914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda07c0c-7263-4923-bfed-a75ed51b9349-istiod-ca-cert\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.306831 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.306804 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "kube-api-access-5jwt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:39:51.306923 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.306843 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 07:39:51.306963 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.306918 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:39:51.306963 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.306925 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy" pod "dda07c0c-7263-4923-bfed-a75ed51b9349" (UID: "dda07c0c-7263-4923-bfed-a75ed51b9349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 07:39:51.406631 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.406596 4914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-podinfo\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.406631 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.406629 4914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-token\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.406828 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.406646 4914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda07c0c-7263-4923-bfed-a75ed51b9349-istio-envoy\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.406828 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.406659 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jwt9\" (UniqueName: \"kubernetes.io/projected/dda07c0c-7263-4923-bfed-a75ed51b9349-kube-api-access-5jwt9\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:39:51.701672 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.701642 4914 generic.go:356] "Generic (PLEG): container finished" podID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerID="b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd" exitCode=0 Nov 01 07:39:51.701873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.701680 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" event={"ID":"dda07c0c-7263-4923-bfed-a75ed51b9349","Type":"ContainerDied","Data":"b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd"} Nov 01 07:39:51.701873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.701683 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" Nov 01 07:39:51.701873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.701717 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj" event={"ID":"dda07c0c-7263-4923-bfed-a75ed51b9349","Type":"ContainerDied","Data":"3af206e17f930d8e06414988b14f21092740f4be3eff82044f2dd9dc4ec4d2ae"} Nov 01 07:39:51.701873 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.701737 4914 scope.go:122] "RemoveContainer" containerID="b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd" Nov 01 07:39:51.711947 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.711930 4914 scope.go:122] "RemoveContainer" containerID="d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862" Nov 01 07:39:51.719877 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.719861 4914 scope.go:122] "RemoveContainer" containerID="b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd" Nov 01 07:39:51.720108 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:51.720086 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd\": container with ID starting with b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd not found: ID does not exist" containerID="b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd" Nov 01 07:39:51.720179 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.720114 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd"} err="failed to get container status \"b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd\": rpc error: code = NotFound desc = could not find container \"b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd\": container with ID starting with b1c94242597eeb21d36afb796b2a158ad6f56af4bfdab0159467dba919944efd not found: ID does not exist" Nov 01 07:39:51.720179 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.720135 4914 scope.go:122] "RemoveContainer" containerID="d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862" Nov 01 07:39:51.720420 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:39:51.720402 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862\": container with ID starting with d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862 not found: ID does not exist" containerID="d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862" Nov 01 07:39:51.720474 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.720426 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862"} err="failed to get container status \"d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862\": rpc error: code = NotFound desc = could not find container \"d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862\": container with ID starting with d72b86893d09097a1cd508362cd8a31de21c5cdf387b1120e37e977f3abd5862 not found: ID does not exist" Nov 01 07:39:51.863694 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.863675 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj"] Nov 01 07:39:51.866911 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:51.866890 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2624b0-openshift-default-5dc45cfc49wjspj"] Nov 01 07:39:52.615333 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:39:52.615301 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" path="/var/lib/kubelet/pods/dda07c0c-7263-4923-bfed-a75ed51b9349/volumes" Nov 01 07:40:07.597111 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:40:07.597067 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:40:24.525754 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.525714 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t"] Nov 01 07:40:24.526440 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526420 4914 cpu_manager.go:414] "containerMap: removing container" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" Nov 01 07:40:24.526531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526442 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" Nov 01 07:40:24.526531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526463 4914 cpu_manager.go:414] "containerMap: removing container" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" Nov 01 07:40:24.526531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526470 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" Nov 01 07:40:24.526531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526483 4914 cpu_manager.go:414] "containerMap: removing container" podUID="1f439840-c085-4206-abfb-5b122ce41314" containerName="maas-api" Nov 01 07:40:24.526531 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526491 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f439840-c085-4206-abfb-5b122ce41314" containerName="maas-api" Nov 01 07:40:24.526787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526587 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" Nov 01 07:40:24.526787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526602 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda07c0c-7263-4923-bfed-a75ed51b9349" containerName="istio-proxy" Nov 01 07:40:24.526787 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.526612 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f439840-c085-4206-abfb-5b122ce41314" containerName="maas-api" Nov 01 07:40:24.529567 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.529545 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.622252 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.622220 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4rw\" (UniqueName: \"kubernetes.io/projected/15a023a8-a1e7-437b-9b09-a8d5916015db-kube-api-access-tl4rw\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.622425 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.622403 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a023a8-a1e7-437b-9b09-a8d5916015db-openshift-service-ca\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.622506 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.622485 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a023a8-a1e7-437b-9b09-a8d5916015db-metrics-certs\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.622651 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.622634 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a023a8-a1e7-437b-9b09-a8d5916015db-maas-api-tls\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.632473 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.632453 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t"] Nov 01 07:40:24.722948 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.722925 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a023a8-a1e7-437b-9b09-a8d5916015db-openshift-service-ca\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.723061 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.722965 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a023a8-a1e7-437b-9b09-a8d5916015db-metrics-certs\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.723061 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.722985 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a023a8-a1e7-437b-9b09-a8d5916015db-maas-api-tls\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.723061 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.723008 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4rw\" (UniqueName: \"kubernetes.io/projected/15a023a8-a1e7-437b-9b09-a8d5916015db-kube-api-access-tl4rw\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.723679 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.723657 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a023a8-a1e7-437b-9b09-a8d5916015db-openshift-service-ca\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.725602 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.725579 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a023a8-a1e7-437b-9b09-a8d5916015db-metrics-certs\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.725683 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.725645 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15a023a8-a1e7-437b-9b09-a8d5916015db-maas-api-tls\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.779230 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.779178 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4rw\" (UniqueName: \"kubernetes.io/projected/15a023a8-a1e7-437b-9b09-a8d5916015db-kube-api-access-tl4rw\") pod \"maas-api-65578dc8d4-s2p7t\" (UID: \"15a023a8-a1e7-437b-9b09-a8d5916015db\") " pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:24.841920 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:24.841894 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:25.210430 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:25.210403 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t"] Nov 01 07:40:25.212361 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:40:25.212330 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a023a8_a1e7_437b_9b09_a8d5916015db.slice/crio-e99e2d94e11f1e0d657a5c7e88e8fbc62cb10d573b6c9825543632cd50df70a2 WatchSource:0}: Error finding container e99e2d94e11f1e0d657a5c7e88e8fbc62cb10d573b6c9825543632cd50df70a2: Status 404 returned error can't find the container with id e99e2d94e11f1e0d657a5c7e88e8fbc62cb10d573b6c9825543632cd50df70a2 Nov 01 07:40:25.877672 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:25.877605 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" event={"ID":"15a023a8-a1e7-437b-9b09-a8d5916015db","Type":"ContainerStarted","Data":"e99e2d94e11f1e0d657a5c7e88e8fbc62cb10d573b6c9825543632cd50df70a2"} Nov 01 07:40:26.884232 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:26.884192 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" event={"ID":"15a023a8-a1e7-437b-9b09-a8d5916015db","Type":"ContainerStarted","Data":"d8129fa4063cf2e6bef369d7d49fb1ee6e3fc0e82a17db9daa6006d6fc3a806b"} Nov 01 07:40:26.884678 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:26.884319 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:27.001186 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:27.001132 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" podStartSLOduration=2.226463504 podStartE2EDuration="3.00111795s" totalImagesPullingTime="774.654446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:40:26.999179328 +0000 UTC m=+5442.551968522" watchObservedRunningTime="2026-11-01 07:40:27.00111795 +0000 UTC m=+5442.553907144" Nov 01 07:40:32.895541 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:32.895515 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65578dc8d4-s2p7t" Nov 01 07:40:37.603174 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:40:37.603125 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:40:41.149184 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:41.149148 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-664qq"] Nov 01 07:40:41.149733 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:41.149374 4914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" podUID="686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" containerName="maas-api" containerID="cri-o://5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88" gracePeriod=30 Nov 01 07:40:42.459635 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.459330 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 07:40:42.526693 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.526661 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs\") pod \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " Nov 01 07:40:42.526854 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.526800 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls\") pod \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " Nov 01 07:40:42.526930 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.526856 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca\") pod \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " Nov 01 07:40:42.527001 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.526941 4914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt\" (UniqueName: \"kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt\") pod \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\" (UID: \"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999\") " Nov 01 07:40:42.527494 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.527407 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca" pod "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" (UID: "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 07:40:42.529826 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.529786 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt" pod "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" (UID: "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999"). InnerVolumeSpecName "kube-api-access-bc5tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 07:40:42.529938 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.529839 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs" pod "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" (UID: "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:40:42.529938 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.529925 4914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls" pod "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" (UID: "686b44ef-2f6c-4fe8-a62e-5d2d8e45f999"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 07:40:42.628169 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.628131 4914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-maas-api-tls\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:42.628169 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.628170 4914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-openshift-service-ca\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:42.628436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.628185 4914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5tt\" (UniqueName: \"kubernetes.io/projected/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-kube-api-access-bc5tt\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:42.628436 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.628202 4914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999-metrics-certs\") on node \"ip-10-0-130-54.us-west-1.compute.internal\" DevicePath \"\"" Nov 01 07:40:42.970820 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.970776 4914 generic.go:356] "Generic (PLEG): container finished" podID="686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" containerID="5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88" exitCode=0 Nov 01 07:40:42.970995 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.970873 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" event={"ID":"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999","Type":"ContainerDied","Data":"5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88"} Nov 01 07:40:42.970995 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.970922 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" event={"ID":"686b44ef-2f6c-4fe8-a62e-5d2d8e45f999","Type":"ContainerDied","Data":"202d5870355886f4995e4c258430b95ca3fc293b303c2738ae2220a74dc4aaa5"} Nov 01 07:40:42.970995 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.970953 4914 scope.go:122] "RemoveContainer" containerID="5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88" Nov 01 07:40:42.971195 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.971095 4914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb4d85776-664qq" Nov 01 07:40:42.984027 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.983999 4914 scope.go:122] "RemoveContainer" containerID="5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88" Nov 01 07:40:42.984402 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:40:42.984366 4914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88\": container with ID starting with 5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88 not found: ID does not exist" containerID="5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88" Nov 01 07:40:42.984508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:42.984416 4914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88"} err="failed to get container status \"5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88\": rpc error: code = NotFound desc = could not find container \"5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88\": container with ID starting with 5aec1b56d2f9892ced622a97442da11b29a914356329e55c9c0fd6497af10f88 not found: ID does not exist" Nov 01 07:40:43.175388 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:43.175345 4914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-664qq"] Nov 01 07:40:43.175890 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:43.175869 4914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb4d85776-664qq"] Nov 01 07:40:44.637535 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:40:44.637498 4914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" path="/var/lib/kubelet/pods/686b44ef-2f6c-4fe8-a62e-5d2d8e45f999/volumes" Nov 01 07:41:07.591196 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:41:07.591157 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:41:37.591804 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:41:37.591767 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:42:00.723355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:42:00.722396 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-vktw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:42:00.723355 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:42:00.722488 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-vktw8" podUID="3f0dea1e-43ae-409a-9eeb-02badf63542c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:42:07.591303 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:42:07.591247 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:42:37.593802 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:42:37.593761 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:43:07.741318 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:43:07.736304 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:43:31.781356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:43:31.779427 4914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-9tp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 07:43:31.781356 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:43:31.779506 4914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-9tp28" podUID="c97e7d90-1e4f-4445-84e6-6198f8bd3024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 07:43:37.595101 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:43:37.595062 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:44:06.135485 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:06.135451 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575f85f67b-6vdks_52b6aac2-2f3b-48e1-b1b1-f0445358abe8/manager/0.log" Nov 01 07:44:07.592379 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:44:07.592345 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:44:08.225243 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:08.225216 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8p824_12e2413a-0690-4528-8f90-94f6bd824f63/server/0.log" Nov 01 07:44:10.121955 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:10.121925 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65578dc8d4-s2p7t_15a023a8-a1e7-437b-9b09-a8d5916015db/maas-api/0.log" Nov 01 07:44:17.479952 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:17.479921 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6r85z_fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63/discovery/0.log" Nov 01 07:44:17.837585 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:17.837512 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cbf89484-6x6nh_845b96c1-0d3f-4af1-affa-5afb60fbebef/kube-auth-proxy/0.log" Nov 01 07:44:19.196762 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:19.196733 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-9tp28_c97e7d90-1e4f-4445-84e6-6198f8bd3024/istio-proxy/0.log" Nov 01 07:44:19.844778 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:19.844742 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-krv87_430c5975-a53c-4dde-814a-6661ad4e71b5/istio-proxy/0.log" Nov 01 07:44:20.629692 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:20.629663 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-qpmzx_f393dcd3-3bd8-430f-aa5c-67af99441bdf/istio-proxy/0.log" Nov 01 07:44:20.959518 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:20.959447 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-vktw8_3f0dea1e-43ae-409a-9eeb-02badf63542c/istio-proxy/0.log" Nov 01 07:44:21.482960 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:21.482930 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5b4cf47bfb-xjj8z_ec5403c5-bc63-43e7-8702-eebf3ffd6cd9/payload-pre-processing/0.log" Nov 01 07:44:21.696697 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:21.696662 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6759564b6c-wpg92_402eb62b-368e-447c-be22-a1e68493ad7c/payload-processing/0.log" Nov 01 07:44:21.925855 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:21.925831 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b77c8f77-457bj_cc06a215-884f-40b1-90fc-208bcc319a76/router/0.log" Nov 01 07:44:23.283111 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:23.283079 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-znq7h_e78c1a4d-d1ee-480c-87ee-1491bde3c1c1/main/0.log" Nov 01 07:44:23.559766 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:23.559686 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-gzgw9_77908dca-3be0-4063-b49e-21b1fff1562c/main/0.log" Nov 01 07:44:23.776397 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:23.776371 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-pmrj9_cb9ae92a-f1fa-4a1b-aa07-9e0ebd8113ef/main/0.log" Nov 01 07:44:24.270759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:24.270735 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hjb2f_24ae932e-1d7d-4e43-9e2c-270cbff00cf9/main/0.log" Nov 01 07:44:35.676055 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.676021 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kd9k9/must-gather-lt2cm"] Nov 01 07:44:35.676444 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.676430 4914 cpu_manager.go:414] "containerMap: removing container" podUID="686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" containerName="maas-api" Nov 01 07:44:35.676444 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.676440 4914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" containerName="maas-api" Nov 01 07:44:35.676520 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.676511 4914 memory_manager.go:365] "RemoveStaleState removing state" podUID="686b44ef-2f6c-4fe8-a62e-5d2d8e45f999" containerName="maas-api" Nov 01 07:44:35.679665 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.679643 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.733428 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.733381 4914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kd9k9\"/\"default-dockercfg-hbgcf\"" Nov 01 07:44:35.773802 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.773780 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c054a95-cae9-474c-9857-b126a831f340-must-gather-output\") pod \"must-gather-lt2cm\" (UID: \"4c054a95-cae9-474c-9857-b126a831f340\") " pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.773917 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.773810 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w8r\" (UniqueName: \"kubernetes.io/projected/4c054a95-cae9-474c-9857-b126a831f340-kube-api-access-g5w8r\") pod \"must-gather-lt2cm\" (UID: \"4c054a95-cae9-474c-9857-b126a831f340\") " pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.783071 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.783052 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kd9k9\"/\"kube-root-ca.crt\"" Nov 01 07:44:35.783187 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.783147 4914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kd9k9\"/\"openshift-service-ca.crt\"" Nov 01 07:44:35.784922 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.784896 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/must-gather-lt2cm"] Nov 01 07:44:35.875009 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.874986 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c054a95-cae9-474c-9857-b126a831f340-must-gather-output\") pod \"must-gather-lt2cm\" (UID: \"4c054a95-cae9-474c-9857-b126a831f340\") " pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.875112 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.875019 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w8r\" (UniqueName: \"kubernetes.io/projected/4c054a95-cae9-474c-9857-b126a831f340-kube-api-access-g5w8r\") pod \"must-gather-lt2cm\" (UID: \"4c054a95-cae9-474c-9857-b126a831f340\") " pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.875403 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.875385 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4c054a95-cae9-474c-9857-b126a831f340-must-gather-output\") pod \"must-gather-lt2cm\" (UID: \"4c054a95-cae9-474c-9857-b126a831f340\") " pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.931655 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.931593 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w8r\" (UniqueName: \"kubernetes.io/projected/4c054a95-cae9-474c-9857-b126a831f340-kube-api-access-g5w8r\") pod \"must-gather-lt2cm\" (UID: \"4c054a95-cae9-474c-9857-b126a831f340\") " pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:35.993221 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:35.993184 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/must-gather-lt2cm" Nov 01 07:44:36.369568 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:36.369545 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/must-gather-lt2cm"] Nov 01 07:44:36.371751 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:44:36.371721 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c054a95_cae9_474c_9857_b126a831f340.slice/crio-57c70ed941caea6c2e282dc138fb9c6b42f7229c761c308b79ec4db37e310ddd WatchSource:0}: Error finding container 57c70ed941caea6c2e282dc138fb9c6b42f7229c761c308b79ec4db37e310ddd: Status 404 returned error can't find the container with id 57c70ed941caea6c2e282dc138fb9c6b42f7229c761c308b79ec4db37e310ddd Nov 01 07:44:36.373467 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:36.373450 4914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 07:44:37.293982 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:37.293942 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/must-gather-lt2cm" event={"ID":"4c054a95-cae9-474c-9857-b126a831f340","Type":"ContainerStarted","Data":"57c70ed941caea6c2e282dc138fb9c6b42f7229c761c308b79ec4db37e310ddd"} Nov 01 07:44:37.598972 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:44:37.598908 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:44:38.300042 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:38.300006 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/must-gather-lt2cm" event={"ID":"4c054a95-cae9-474c-9857-b126a831f340","Type":"ContainerStarted","Data":"c78e8c824e1b3ae7822142700a2ee914d72db424f9c5b0f46109815b68d56032"} Nov 01 07:44:38.300042 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:38.300049 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/must-gather-lt2cm" event={"ID":"4c054a95-cae9-474c-9857-b126a831f340","Type":"ContainerStarted","Data":"06daf6f5957932748b1b70f3522b8ed7d8680fdf11e0b0d2729a0b4e0f4932f1"} Nov 01 07:44:38.414851 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:38.414797 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kd9k9/must-gather-lt2cm" podStartSLOduration=2.2224358889999998 podStartE2EDuration="3.414781995s" totalImagesPullingTime="1.192346106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 07:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:44:38.413734865 +0000 UTC m=+5693.966524059" watchObservedRunningTime="2026-11-01 07:44:38.414781995 +0000 UTC m=+5693.967571192" Nov 01 07:44:44.613884 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:44.613851 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rj4mc_1d8e93ea-2cf0-4f69-ba21-ddf7c95fec9a/global-pull-secret-syncer/0.log" Nov 01 07:44:45.373294 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:45.372597 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:44:45.425299 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:45.420747 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-fqwwj_1b4526c0-30f7-4558-b9e6-4a949e19e8fb/console-operator/2.log" Nov 01 07:44:46.390301 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:46.390255 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6tflw_dd901c17-d465-4a03-b320-73be6ff3d0ce/connection-checker/0.log" Nov 01 07:44:46.675526 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:46.675363 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575f85f67b-6vdks_52b6aac2-2f3b-48e1-b1b1-f0445358abe8/manager/0.log" Nov 01 07:44:46.912078 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:46.912036 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cwffg_8079984d-980a-4287-826e-91194cf07e9e/connection-checker/0.log" Nov 01 07:44:47.334297 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:47.333924 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lpzzd_99cae588-1b8c-4972-bda1-781a3edb687a/connection-checker/0.log" Nov 01 07:44:48.248704 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:48.248674 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l576t_e862b42b-a7df-440b-9170-89c0de89c451/konnectivity-agent/0.log" Nov 01 07:44:48.422175 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:48.417239 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8p824_12e2413a-0690-4528-8f90-94f6bd824f63/server/0.log" Nov 01 07:44:49.072552 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:49.072511 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-54.us-west-1.compute.internal_2b0505d995be0782193889599bca279d/haproxy/0.log" Nov 01 07:44:53.425206 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:53.425164 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575f85f67b-6vdks_52b6aac2-2f3b-48e1-b1b1-f0445358abe8/manager/0.log" Nov 01 07:44:54.296181 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.295821 4914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths"] Nov 01 07:44:54.300046 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.300019 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.391878 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.391840 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths"] Nov 01 07:44:54.393389 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.393353 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-podres\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.393589 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.393561 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cjc\" (UniqueName: \"kubernetes.io/projected/37f0f5f8-f902-445d-a76d-4c4d9492f107-kube-api-access-72cjc\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.393645 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.393610 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-sys\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.393707 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.393686 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-lib-modules\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.393759 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.393747 4914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-proc\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.494796 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.494753 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cjc\" (UniqueName: \"kubernetes.io/projected/37f0f5f8-f902-445d-a76d-4c4d9492f107-kube-api-access-72cjc\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.494796 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.494790 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-sys\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.494812 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-lib-modules\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.494880 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-sys\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.494904 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-lib-modules\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.494980 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-proc\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.495026 4914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-podres\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.495060 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-proc\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.495380 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.495139 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37f0f5f8-f902-445d-a76d-4c4d9492f107-podres\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.553508 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.553444 4914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cjc\" (UniqueName: \"kubernetes.io/projected/37f0f5f8-f902-445d-a76d-4c4d9492f107-kube-api-access-72cjc\") pod \"perf-node-gather-daemonset-5pths\" (UID: \"37f0f5f8-f902-445d-a76d-4c4d9492f107\") " pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.612034 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.612002 4914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:54.985256 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:54.985227 4914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths"] Nov 01 07:44:54.987818 ip-10-0-130-54 kubenswrapper[4914]: W1101 07:44:54.987787 4914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37f0f5f8_f902_445d_a76d_4c4d9492f107.slice/crio-0b52f353098d54d1fbaa53393aa86215fdc2e42f700d6efd1d4c9a0011a7a827 WatchSource:0}: Error finding container 0b52f353098d54d1fbaa53393aa86215fdc2e42f700d6efd1d4c9a0011a7a827: Status 404 returned error can't find the container with id 0b52f353098d54d1fbaa53393aa86215fdc2e42f700d6efd1d4c9a0011a7a827 Nov 01 07:44:55.117688 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:55.117663 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8p824_12e2413a-0690-4528-8f90-94f6bd824f63/server/0.log" Nov 01 07:44:55.487062 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:55.487022 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" event={"ID":"37f0f5f8-f902-445d-a76d-4c4d9492f107","Type":"ContainerStarted","Data":"ea1aa85c301f2add858d2ea1c36b7206a6f947cdcb4bc4cd1e127adba8cf1702"} Nov 01 07:44:55.487231 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:55.487093 4914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" event={"ID":"37f0f5f8-f902-445d-a76d-4c4d9492f107","Type":"ContainerStarted","Data":"0b52f353098d54d1fbaa53393aa86215fdc2e42f700d6efd1d4c9a0011a7a827"} Nov 01 07:44:55.487317 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:55.487246 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:44:55.600536 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:44:55.600468 4914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" podStartSLOduration=1.600448219 podStartE2EDuration="1.600448219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 07:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 07:44:55.599794947 +0000 UTC m=+5711.152584143" watchObservedRunningTime="2026-11-01 07:44:55.600448219 +0000 UTC m=+5711.153237414" Nov 01 07:45:00.060838 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:00.060758 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575f85f67b-6vdks_52b6aac2-2f3b-48e1-b1b1-f0445358abe8/manager/0.log" Nov 01 07:45:01.503098 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:01.503067 4914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kd9k9/perf-node-gather-daemonset-5pths" Nov 01 07:45:01.935064 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:01.935033 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8p824_12e2413a-0690-4528-8f90-94f6bd824f63/server/0.log" Nov 01 07:45:03.476585 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:03.476557 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-962j5_cfef5efe-9658-4dbe-a3b5-cb484fec7962/cluster-monitoring-operator/0.log" Nov 01 07:45:07.591658 ip-10-0-130-54 kubenswrapper[4914]: E1101 07:45:07.591622 4914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zf9qt" containerName="extractor" Nov 01 07:45:08.446386 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:08.446353 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5cb4_ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9/node-exporter/0.log" Nov 01 07:45:08.456768 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:08.456738 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6r85z_fbe9d4c1-5b96-4ecc-8b28-a0a66b536e63/discovery/0.log" Nov 01 07:45:08.585570 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:08.585544 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cbf89484-6x6nh_845b96c1-0d3f-4af1-affa-5afb60fbebef/kube-auth-proxy/0.log" Nov 01 07:45:08.890416 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:08.890378 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5cb4_ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9/kube-rbac-proxy/0.log" Nov 01 07:45:09.302829 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:09.302753 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x5cb4_ea5cbfb1-5a87-4d96-b344-6f5aab9f37b9/init-textfile/0.log" Nov 01 07:45:09.554990 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:09.554906 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-9tp28_c97e7d90-1e4f-4445-84e6-6198f8bd3024/istio-proxy/0.log" Nov 01 07:45:09.982166 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:09.982134 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-krv87_430c5975-a53c-4dde-814a-6661ad4e71b5/istio-proxy/0.log" Nov 01 07:45:10.620112 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:10.620081 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-qpmzx_f393dcd3-3bd8-430f-aa5c-67af99441bdf/istio-proxy/0.log" Nov 01 07:45:10.832481 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:10.832449 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-vktw8_3f0dea1e-43ae-409a-9eeb-02badf63542c/istio-proxy/0.log" Nov 01 07:45:11.256468 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:11.256431 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5b4cf47bfb-xjj8z_ec5403c5-bc63-43e7-8702-eebf3ffd6cd9/payload-pre-processing/0.log" Nov 01 07:45:11.465583 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:11.465556 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6759564b6c-wpg92_402eb62b-368e-447c-be22-a1e68493ad7c/payload-processing/0.log" Nov 01 07:45:11.682570 ip-10-0-130-54 kubenswrapper[4914]: I1101 07:45:11.682542 4914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b77c8f77-457bj_cc06a215-884f-40b1-90fc-208bcc319a76/router/0.log"